NISQ (Noisy Intermediate-Scale Quantum) devices represent the current generation of quantum processors, typically containing 50 to a few hundred qubits, but without full error correction. While they hold great promise for early quantum advantage, these devices are still error-prone and require careful benchmarking to assess performance, reliability, and usefulness in practical applications.
Benchmarking NISQ devices is essential for:
- Understanding device capabilities,
- Comparing across platforms (e.g., IBM vs IonQ),
- Identifying bottlenecks in algorithm execution,
- Informing algorithm design for maximum effectiveness.
1. What Is Benchmarking in Quantum Computing?
Benchmarking involves quantitative measurement of a quantum system’s behavior across different performance indicators. For NISQ devices, benchmarking doesn’t only mean running quantum algorithms—it means measuring how well the system can perform them under noisy, real-world conditions.
Key benchmarking goals include:
- Estimating noise levels,
- Evaluating gate and readout fidelity,
- Testing how deep or wide circuits can run reliably,
- Assessing cross-talk and device stability over time.
2. Categories of Benchmarking Metrics
A. Low-Level Benchmarks (Hardware-Level)
These metrics assess the basic operations of quantum devices:
- Qubit Coherence Times (T₁ and T₂): How long a qubit retains information.
- Gate Fidelity: Accuracy of single-qubit and multi-qubit operations.
- Readout Fidelity: Accuracy in measuring a qubit’s final state.
- Gate Speed: Time taken for gate operations.
- Crosstalk and Noise Propagation: Interaction between qubits during simultaneous operations.
Measurement Techniques:
- Randomized benchmarking,
- Quantum process tomography,
- Cross-entropy benchmarking.
B. Mid-Level Benchmarks (System-Level)
These test how well the entire system performs on specific standard tasks.
- Quantum Volume (QV): Measures the largest random circuit the device can execute successfully.
- Cycle Benchmarking: Tests fidelity of repeated operation cycles.
- Heavy Output Generation (HOG): Measures how often the quantum system produces “heavier” (high-probability) outputs compared to classical predictions.
Purpose: Reveals the combined effect of gate errors, crosstalk, and decoherence.
C. High-Level Benchmarks (Application-Level)
These benchmarks assess how well a NISQ device performs practical workloads like:
- Variational Quantum Eigensolvers (VQE),
- Quantum Approximate Optimization Algorithm (QAOA),
- Quantum Machine Learning (QML) models,
- Simulation of small molecules or materials.
Output: Comparison against known classical baselines.
3. Benchmarking Techniques and Tools
A. Randomized Benchmarking (RB)
- Applies sequences of random Clifford gates,
- Measures how error accumulates with sequence length,
- Reduces sensitivity to state preparation and measurement (SPAM) errors.
Advantage: Scalable to many qubits.
B. Quantum Volume (QV)
- Measures the maximum width × depth circuit with success probability > 2/3.
- Accounts for:
- Connectivity,
- Crosstalk,
- Gate fidelity,
- Compiler efficiency.
Used By: IBM, Honeywell, Amazon Braket.
C. Cross-Entropy Benchmarking
- Compares output of random quantum circuits to ideal output distributions.
- Often used in quantum supremacy experiments (e.g., Google’s Sycamore).
Metric: Linear cross-entropy fidelity.
D. Cycle Benchmarking
- Evaluates gate fidelity under repeated application.
- Well-suited for multi-qubit operations and real-use conditions.
E. Algorithmic Benchmarking
- Run real-world quantum algorithms and evaluate:
- Success rate,
- Fidelity of output,
- Comparison with classical approximations.
Algorithms: QAOA, VQE, Grover’s, etc.
4. Benchmarks Across Hardware Platforms
Each hardware platform exhibits unique benchmarking behavior:
| Platform | Strength in Benchmarking | Example Metric Used |
|---|---|---|
| IBM (Superconducting) | Quantum Volume, RB | QV up to 128 |
| IonQ (Trapped Ion) | Algorithmic fidelity, low noise | VQE benchmarks |
| Xanadu (Photonic) | Quantum ML, circuit depth tests | Interferometer performance |
| Quantinuum | QV, cross-entropy | High multi-qubit fidelity |
5. Challenges in Benchmarking NISQ Devices
A. Device Variability
- Qubit-to-qubit performance varies significantly.
- Requires per-device calibration and testing.
B. Limited Circuit Depth
- Noise grows with circuit depth.
- Benchmarking must simulate realistic, shallow-depth circuits.
C. Noisy Measurements
- SPAM errors can bias results if not corrected.
- Must be accounted for in benchmarking interpretation.
D. Compilation Overhead
- Circuit transpilation affects depth and width.
- Benchmarks must include compiler performance to be realistic.
E. Temporal Drift
- Device performance fluctuates over time.
- Benchmarks should be repeated across time spans.
6. Importance of Benchmarking for Developers
- Algorithm Design: Knowing qubit limits helps shape efficient algorithms.
- Noise-Aware Compilation: Select better qubit paths to avoid poor-performing regions.
- Hardware Selection: Choose the right platform for the task at hand.
- Trust & Certification: Independent benchmarking helps validate vendor claims.
7. Best Practices
- Benchmark before every major workload.
- Use both hardware and application-level benchmarks.
- Automate with tools like Qiskit Ignis, Cirq, or Braket SDKs.
- Log historical performance to detect drift or degradation.
- Use noise-aware simulators for comparative testing.
8. Future of Benchmarking
In the coming years, benchmarking NISQ systems will evolve to include:
- Standardized metrics across vendors,
- AI-driven adaptive benchmarking, selecting circuits that stress weaknesses,
- Real-time benchmarking dashboards embedded in quantum development environments,
- Hardware-agnostic abstraction layers so developers don’t need to understand every device quirk.