SR Quantum Backend Bench

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project follows Semantic Versioning.

[Unreleased]

Added

  • GitHub Pages workflow and portfolio-style documentation landing page for the project website.
  • RESULTS.md with curated benchmark tables, reproduction commands, and generated plot assets for the website.
  • Website link in the README.

[0.1.7] - 2026-04-29

Added

  • QAOA MaxCut benchmark for single-layer line and ring graph optimization workloads.
  • Packaged comparison presets for runtime, noise, algorithmic, and SDK smoke studies.
  • quantum-bench preset commands for listing, exporting, and running packaged presets.
  • quantum-bench report for Markdown reports from saved JSON, result bundles, or CSV exports.
  • Experiment manifest outputs.report support for Markdown report generation.
  • Qiskit Aer depolarizing-noise injection for noisy benchmark wrappers.

Changed

  • Backend discovery and result metadata now report Qiskit Aer depolarizing noise support.
  • The standard suite now includes a small QAOA MaxCut optimization workload.
  • The all extra now targets the practical Python-only comparison stack; full includes heavyweight and external-runtime-backed SDK extras.
  • Optional backend tests are marked by Python SDK, heavyweight SDK, and external runtime requirements.

[0.1.6] - 2026-04-29

Added

  • quantum-bench doctor for local integration diagnostics and install hints.
  • CLI validation for positive --shots and --repeats, probability-bounded success thresholds, and noise levels.
  • Backend runtime metadata in result rows, including transpilation/runtime caveats, external-process usage, local-only status, noise support, and backend package versions.
  • Best-effort deterministic simulator seeding for backends that expose seed controls.
  • Packaging regression coverage for required source-distribution documentation.
  • Python 3.11 and 3.12 CI coverage.
  • Backend-selection guidance, result-interpretation notes, and a reproducibility checklist in the documentation.

Changed

  • CI now runs the main test/build job across the supported Python version matrix.

[0.1.5] - 2026-04-29

Added

  • THEORY.md with conceptual background for qubits, circuits, shots, distributions, metrics, noise models, benchmark families, and backend comparability.
  • Tables of contents for long-form and reference Markdown documentation.
  • Public benchmark factory helpers for CLI-style benchmark configuration.

Changed

  • Experiment manifest execution now builds benchmarks through shared core factory helpers instead of importing CLI internals.
  • The all optional dependency extra now includes qbraid.
  • Optional-backend CI coverage now includes CUDA-Q and pyQuil smoke jobs, with CUDA-Q marked experimental.

Fixed

  • run_benchmark now rejects invalid shots values before dispatching to backend adapters.

[0.1.4] - 2026-04-29

Added

  • quantum-bench diff for comparing saved JSON or CSV result files with metric thresholds.
  • Public result-diff helpers for loading saved outputs, comparing metric deltas, and formatting diff tables.

Fixed

  • pyQuil QVM discovery now requires local qvm and quilc executables in addition to the Python package.

[0.1.3] - 2026-04-24

Added

  • Distribution, heatmap, noise-quality, and suite runtime plot outputs.
  • Plot gallery and circuit diagram export examples.
  • Bra-ket notation for computational basis states in plots.
  • Backend legend for suite runtime plots.
  • GHZ plot-gallery noise sweep now spans 0% to 5%.
  • Bernstein-Vazirani benchmark with configurable hidden bitstring.
  • Deutsch-Jozsa benchmark with constant and linear balanced oracle modes.
  • Quantum-volume-style random layered benchmark.
  • CLI and suite coverage for the new benchmark families.
  • quantum-bench list and quantum-bench info discovery commands.
  • Suite dry-run/listing support with JSON suite manifest export.
  • Standardized result metadata fields for benchmark family, case labels, depth, seed, oracle type, and noise level.
  • Optional Qiskit Aer, CUDA-Q, pyQuil QVM, and QuTiP local execution adapters.
  • Optional qBraid and Q# integration discovery extras.
  • Experiment manifest runner with JSON/YAML loading, output export, and bundled research-style manifests.
  • Runtime repeat statistics and environment capture in result metadata.
  • Result schema and benchmark methodology documentation.
  • Research helper examples for capability matrices, manifest generation, repeated-runtime analysis, schema inspection, noise manifests, and Markdown reports.
  • examples/README.md with a recommended research example workflow and expected artifacts.
  • quantum-bench validate for installed-backend correctness and regression checks.
  • quantum-bench recommend for backend selection by use case.
  • Problem statement, limitations, and citation metadata.
  • Example scripts for oracle benchmarks, quantum-volume-style execution, and suite artifact export.
  • Tests for new benchmark builders, CLI execution, and Cirq Bernstein-Vazirani correctness.

Changed

  • CSV exports now include case labels and benchmark families.
  • Plot labels now use standardized result case labels.
  • Backend CLI choices are now driven by the backend registry.
  • CLI result tables and CSV exports include repeat-aware runtime fields.

Fixed

  • Benchmark validation errors now report clearer expected lengths and invalid parameter constraints.

[0.1.2] - 2026-04-24

Added

  • CI workflow for formatting, linting, tests, build, and distribution checks.
  • Optional backend extras for Cirq, PennyLane, Braket, pytket, plotting, and all integrations.
  • --summary result ranking output for CLI benchmark result commands.
  • --save-csv export support for benchmark results.
  • quantum-bench suite command with smoke, standard, and scaling presets.
  • Public exports for suite and summary helpers.
  • Additional tests for backend bitstring consistency, Grover success metrics, QFT structure, noise metadata, suites, CSV export, summary output, and result ranking.

Changed

  • Split backend SDKs and plotting dependencies into optional extras.
  • Removed tracked local Codex metadata from the repository.

Fixed

  • Missing backend and plotting dependencies now report the matching optional extra install command.

[0.1.1] - 2026-04-23

Added

  • Native circuit drawing through Cirq, PennyLane, Amazon Braket, and pytket.
  • quantum-bench draw CLI command for rendering benchmark circuits and optionally saving diagrams.
  • USAGE.md with task-oriented CLI, Python API, result schema, and development workflow guidance.
  • MANIFEST.in, .gitignore, and .gitattributes for cleaner source distributions and repository hygiene.

Changed

  • Updated README badges, installation guidance, release build instructions, and circuit drawing examples.
  • Configured writable local runtime cache directories for matplotlib and Numba in Codespaces-style environments.
  • Updated PennyLane execution to use qml.set_shots(...).

Fixed

  • Replaced deprecated or removed pytket Circuit.optypes() usage with command-based gate histogram analysis.
  • Avoided Braket LocalSimulator Numba cache failures in restricted local environments.
  • Removed generated artifacts, egg-info metadata, and Python bytecode caches from git tracking.

[0.1.0] - 2026-04-23

Initial public release of quantum-backend-bench.

Added

  • Backend-agnostic benchmarking package structure with a shared BenchmarkSpec abstraction and a unified run_benchmark(...) API.
  • Local execution backends for Cirq, PennyLane, and Amazon Braket LocalSimulator, with no cloud credentials required.
  • pytket-based structural analysis for circuit depth, gate counts, and two-qubit gate metrics.
  • Built-in benchmark builders for GHZ, QFT, random circuits, Grover search, Hamiltonian simulation, and noise sensitivity sweeps.
  • Standardized metric calculation for runtime, measurement distributions, success probability, and total variation distance.
  • quantum-bench CLI with run, compare, and noise-sweep commands.
  • JSON export helpers, matplotlib plotting utilities, and example scripts for backend comparison and noise benchmarking.
  • Automated tests covering benchmark builders, backend interfaces, metric helpers, and CLI execution.
  • GitHub Codespaces development container configuration for Python 3.11+.
  • PyPI publish workflow triggered from version tags.