pulses. The success of implementing a particular gate can be expressed by various metrics such as the average gate fidelity, the diamond distance, and the unitarity. We analyze these metrics of gate pulses for a system of two superconducting transmon qubits coupled by a resonator, a system inspired by the architecture of the IBM Quantum Experience. The metrics are obtained by numerical solution of the time-dependent Schr\“odinger equation of the transmon system. We find that the metrics reflect systematic errors that are most pronounced for echoed cross-resonance gates, but that none of the studied metrics can reliably predict the performance of a gate when used repeatedly in a quantum algorithm.