SFQ-based gate fidelity was limited by quasiparticle (QP) poisoning induced by the dissipative on-chip SFQ driver circuit. In this work, we introduce a multi-chip module architecture to suppress phonon-mediated QP poisoning. Here, the SFQ elements and qubits are fabricated on separate chips that are joined with In bump bonds. We use interleaved randomized benchmarking to characterize the fidelity of SFQ-based gates, and we demonstrate an error per Clifford gate of 1.2(1)%, an order-of-magnitude reduction over the gate error achieved in the initial realization of SFQ-based qubit control. We use purity benchmarking to quantify the contribution of incoherent error at 0.96(2)%; we attribute this error to photon-mediated QP poisoning mediated by the resonant mm-wave antenna modes of the qubit and SFQ-qubit coupler. We anticipate that a straightforward redesign of the SFQ driver circuit to limit the bandwidth of the SFQ pulses will eliminate this source of infidelity, allowing SFQ-based gates with fidelity approaching theoretical limits, namely 99.9% for resonant sequences and 99.99% for more complex pulse sequences involving variable pulse-to-pulse separation.