up the platform is the requirement to communicate on demand the information between the cavities. It has been recently demonstrated that a tunable bilinear interaction between two cavities can be realized by coupling them to a bichromatically-driven transmon ancilla, which allows swapping and interfering the multi-photon states of the cavities [Gao et al., Phys. Rev. X 8, 021073(2018)]. Here, we explore both theoretically and experimentally the regime of relatively strong drives on the ancilla needed to achieve fast SWAP gates but which can also lead to undesired non-perturbative effects that lower the SWAP fidelity. We develop a theoretical formalism based on linear response theory that allows one to calculate the rate of ancilla-induced interaction, decay and frequency shift of the cavities in terms of a susceptibility matrix. We treat the drives non-perturbatively using Floquet theory, and find that the interference of the two drives can strongly alter the system dynamics even in the regime where the rotating wave approximation applies. We identify two major sources of infidelity due to ancilla decoherence. i) Ancilla dissipation and dephasing lead to incoherent hopping among ancilla Floquet states, which results in a sudden change of the SWAP rate thereby decohering the operations. ii) The cavities inherit finite decay from the relatively lossy ancilla through the inverse Purcell effect; the effect can be enhanced when the drive-induced AC Stark shift pushes certain ancilla transition frequencies to the vicinity of the cavity frequencies. The theoretical predictions agree quantitatively with the experimental results, paving the way for using the theory for designing and optimizing future experiments.