Linking classical microwave electrical circuits to the optical telecommunication band is at the core of modern communication. Future quantum information networks will require coherentmicrowave-to-optical conversion to link electronic quantum processors and memories via low-loss optical telecommunication networks. Efficient conversion can be achieved with electro-optical modulators operating at the single microwave photon level. In the standard electro-optic modulation scheme this is impossible because both, up- and downconverted, sidebands are necessarily present. Here we demonstrate true single sideband up- or downconversion in a triply resonant whispering gallery mode resonator by explicitly addressing modes with asymmetric free spectral range. Compared to previous experiments, we show a three orders of magnitude improvement of the electro-optical conversion efficiency reaching 0.1% photon number conversion for a 10GHz microwave tone at 0.42mW of optical pump power. The presented scheme is fully compatible with existing superconducting 3D circuit quantum electrodynamics technology and can be used for non-classical state conversion and communication. Our conversion bandwidth is larger than 1MHz and not fundamentally limited.