However, the choice of base material also influences the sensitivity to quasiparticle-induced decoherence. In this study, we investigate quasiparticle tunneling rates in niobium and tantalum-based offset-charge-sensitive qubits. Using a source of thermal radiation, we characterize the sensitivity of either material to infrared radiation and explore the impact of the infrared background through the targeted use of in-line filters in the wiring and ambient infrared absorbers. We identify both radiation channels as significant contributions to decoherence for tantalum but not for niobium qubits and achieve tunneling rates of 100 Hz and 300 Hz for niobium and tantalum respectively upon installation of infrared filters. Additionally, we find a time-dependence in the observed tunneling rates on the scale of days, which we interpret as evidence of slowly cooling, thermally radiating components in the experimental setup. Our findings indicate that continued improvements in coherence times may require renewed attention to radiative backgrounds and experimental setup design, especially when introducing new material platforms.