this problem at temperatures above 1~K, in this study we quantify for the first time the losses driven by magnetic vortices for niobium-based quantum devices operating down to millikelvin temperature, and in the low photon counts regime. By cooling a single interface system a 3-D niobium superconducting cavity in a dilution refrigerator through the superconducting transition temperature in controlled levels of magnetic fields, we isolate the flux-induced losses and quantify the added surface resistance per unit of trapped magnetic flux. Our findings indicate that magnetic flux introduces approximately 2~nΩ/mG at 10~mK and at 6~GHz in high RRR niobium. We find that the decay rate of a 6~GHz niobium cavity at 10~mK which contains a native niobium pentoxide will be dominated by the TLS oxide losses until vortices begin to impact T1 for trapped magnetic field (Btrap) levels of >100~mG. In the absence of the niobium pentoxide, Btrap=~10~mG limits Q0∼~10\textsuperscript{10}, or T1∼~350~ms, highlighting the importance of magnetic shielding and magnetic hygiene in enabling T1>~1~s. We observe that the flux-induced resistance decreases with temperature-yet remains largely field-independent, qualitatively explained by thermal activation of vortices in the flux-flow regime. We present a phenomenological model which captures the salient experimental observations. Scaling our findings to typical transmon qubit dimensions suggests that these 2-D structures could be robust against vortex dissipation up to several hundreds~mG. We are directly addressing vortex losses in transmon qubits made with low RRR Nb films in a separate experimental study.