quantum systems could act as quantum memories for these circuits. A useful quantum memory must be rapidly addressable by qubits, while maintaining superior coherence. We demonstrate a novel superconducting microwave cavity architecture that is highly robust against major sources of loss that are encountered in the engineering of circuit QED systems. The architecture allows for near-millisecond storage of quantum states in a resonator while strong coupling between the resonator and a transmon qubit enables control, encoding, and readout at MHz rates. The observed coherence times constitute an improvement of almost an order of magnitude over those of the best available superconducting qubits. Our design is an ideal platform for studying coherent quantum optics and marks an important step towards hardware-efficient quantum computing with Josephson junction-based quantum circuits.