Z2 symmetry of the quantum Rabi model allow us to build an effective three-level {\Lambda}-structured artificial atom for quantum computation. The proposed physical implementation includes two gradiometric flux qubits and two microwave resonators where single-qubit gates are realized by a two-tone driving on one physical qubit, and a two-qubit gate is achieved with a time-dependent coupling between the field quadratures of both resonators. Our work paves the way for scalable holonomic quantum computation in ultrastrongly coupled systems.