Skip to content

Benchmarking & Diagnostics

Tools for benchmarking and diagnosing failures, covering both the core Rust engine and the Python bindings.

The Tooling Stack

Strict boundaries between measurement tools, to avoid the "Observer Effect" (see Tier 2). Three tiers measure Locus itself; a comparative tier measures it against other detectors.

Comparative tier: cargo xtask sota (published comparisons)

  • Tool: cargo xtask sota (xtask/README.md).
  • Purpose: Reproducible comparison against pinned references — OpenCV aruco 4.10.0 (three corner refiners), aruco_nano 961b18b, AprilTag 3 — on every dataset they can decode (Liu4K, EuRoC, ICRA 2020, the render-tag and board suites). It reports recall, precision, F1, order-preserving and debiased corner error, and latency, and ends with a win table; cargo xtask sota scoreboard aggregates every scored benchmark.
  • Rule: Any published "Locus vs X" number comes from this tool. Publish latency only from --jobs 1 runs on an idle machine. The Python sweep tools (bench sweep / tune / compare-instances) use the unpinned opencv-python wheel from the bench group and are for exploration, not publication.

Tier 1: End-to-End Regression (The Python CLI)

  • Tool: uv run tools/cli.py bench real (ICRA 2020 scenarios or Hugging Face Hub datasets via --hub-config).
  • Purpose: Measures the true wall-clock time from Python memory ingestion, through the FFI boundary, across the Rust math kernels, and back to Python. Also reports recall and pose error (translation RMSE in metres) when ground truth poses are available.
  • Rule: This is the ultimate ground truth for latency. If a micro-optimization doesn't lower this number, it didn't actually work.

Tier 2: Macro-Profiling (Tracy)

  • Tool: tracing-tracy combined with the Tracy GUI profiler (cargo test --features tracy).
  • Purpose: Identifies which pipeline stage is the bottleneck (e.g., proving that decode_batch_soa is taking 10ms while extract_quads is taking 2ms).
  • Rule: Never run Tracy concurrently with JSON loggers or console formatters. The string allocation overhead will pollute the nanosecond lock-free ring buffers.

Tier 3: Micro-Benchmarking (Divan)

  • Tool: cargo bench using the Divan framework in crates/locus-core/benches/.
  • Purpose: Measures the Instructions Per Clock (IPC) and L1 cache utilization of isolated, single-threaded mathematical kernels (like SIMD bilinear sampling).
  • Rule: Run these strictly single-threaded to prevent the OS scheduler or Rayon from thrashing the L1 cache.

Rust Benchmarking (Core Engine)

Source of truth for core engine performance and regressions.

Dataset provisioning

Every external dataset (Hub render suites, ICRA 2020, EuRoC, Liu4K) is pinned in xtask/datasets.toml and fetched into tests/data/ with cargo xtask data:

cargo xtask data list                                  # what exists, where, licence
cargo xtask data fetch icra2020-forward icra2020-circle
cargo xtask data fetch euroc liu4k                     # EuRoC mirror needs `hf auth login` or HF_TOKEN
cargo xtask data fetch hub --subsets all
cargo xtask data verify                                # missing or stale-pin copies -> exit 1

Rust tests and Python loaders read the same tests/data/ paths. LOCUS_ICRA_DATASET_DIR, LOCUS_EUROC_DATASET_DIR and LOCUS_HUB_DATASET_DIR only relocate a dataset — except that the render-tag and distortion Hub suites still require LOCUS_HUB_DATASET_DIR to be set (they skip otherwise).

Regression Suite (ICRA 2020)

  1. Fetch the dataset: cargo xtask data fetch icra2020-forward (plus icra2020-circle / icra2020-random for the extended run).
  2. Run Benchmarks: 
    # Core check (Forward dataset + Fixtures, approx 15s)
cargo test --release --test regression_icra2020 --features bench-internals

# Extended check (Circle, Random, Rotation, approx 1-2 mins)
LOCUS_EXTENDED_REGRESSION=1 cargo test --release --test regression_icra2020 --features bench-internals

# Accurate latency measurement (sequential)
cargo test --release --test regression_icra2020 --features bench-internals -- --test-threads=1

    [!IMPORTANT] --release is mandatory for running regression_icra2020 tests. Running in debug mode is blocked and will panic.

Hub Regression Suite (Hugging Face)

[!IMPORTANT] --release is mandatory for running Hub regression tests. Running in debug mode is extremely slow and will likely timeout in CI or developer environments.

  1. Synchronize Data: Download Hub subsets to the local cache (tests/data/hub_cache/) at the revision pinned in xtask/datasets.toml (cargo xtask data list shows every dataset). With no --subsets the manifest's default render-tag set is fetched; all discovers every config: 

    cargo xtask data fetch hub --subsets all
    Or sync specific subsets: 
    cargo xtask data fetch hub --subsets \
locus_v1_tag36h11_640x480,locus_v1_tag36h11_1280x720,locus_v1_tag36h11_1920x1080,\
locus_v1_tag36h11_3840x2160,charuco_golden_v1_1920x1080,aprilgrid_golden_v1_1920x1080

  2. Run Hub Tests: 

    # Tag-level regression (regression_render_tag)
# Covers the 4-resolution `high_accuracy` baseline plus quad-extraction variants
# (EdLines with None refinement, moments culling) at 720p/1080p.
# Requires LOCUS_HUB_DATASET_DIR to locate the cache.
LOCUS_HUB_DATASET_DIR=tests/data/hub_cache \
  cargo test --release --test regression_render_tag --features bench-internals -- --nocapture

# Board-level regression (regression_board_hub)
# Validates ChAruco and AprilGrid golden datasets.
# Uses workspace-relative tests/data/hub_cache/ automatically — no env var needed.
cargo test --release --test regression_board_hub --features bench-internals -- --nocapture

Logic-Specific Benchs (Micro-benchmarking)

For fine-grained benchmarking of specific components, we use Divan. These are located in crates/locus-core/benches.

# Run all micro-benchmarks
cargo bench

# Run specific micro-benchmark (e.g., real-world data)
cargo bench --bench real_data_bench
cargo bench --bench decoding_real_bench

Mutually Exclusive Telemetry Matrix

Emitting structured JSON logs and capturing high-fidelity Tracy profiles simultaneously skews the nanosecond timings (JSON serialization pollutes Tracy's ring buffers). TELEMETRY_MODE decouples them.

Human Mode (Tracy)

Captures pristine binary traces for GUI analysis. 

# Tracy client is assumed to be running or capturing headlessly
TRACY_NO_INVARIANT_CHECK=1 TELEMETRY_MODE=tracy cargo test --release --test regression_icra2020 --features tracy,bench-internals -- --test-threads=1

Agent/CI Mode (JSON)

Dumps structured pipeline timings to target/profiling/*_events.json for AI analysis and automated regression tracking. 

TELEMETRY_MODE=json cargo test --release --test regression_icra2020 --features bench-internals -- --test-threads=1

Performance Reports

Methodology & durable learnings

  • Engineering Lessons — distilled investigation conclusions (pose covariance, rotation tail & edge refinement, EdLines, recall/quad/ICRA): what was tried, what shipped, what was falsified
  • Benchmarking Lessons — consolidated timeline + architecture/profile/algorithm tradeoffs that should inform future work
  • Micro-Benchmarking Guide — 3-tier validation loop

Point-in-time reports (historical snapshots)

The two 2026-10-04 reports describe the current standard pipeline; the rest are history.

Python Developer CLI

tools/cli.py is the central entry point for high-level evaluations and development tasks.

Data Preparation

Use cargo xtask data fetch … (see Dataset provisioning). tools/cli.py bench prepare is a thin wrapper over the same pinned manifest (ICRA forward + circle and every Hub config).

Real-World Evaluation (ICRA 2020)

Evaluate performance on the ICRA 2020 dataset scenarios (forward, circle): 

# Basic run on Locus
PYTHONPATH=. uv run --group bench tools/cli.py bench real --scenarios forward

# Compare against OpenCV and AprilTag 3
PYTHONPATH=. uv run --group bench tools/cli.py bench real --scenarios forward --compare

Real-World Evaluation (Liu4K)

Liu4K is a 924-image, roughly 4K real-photo ArUco set released with the aruco_nano paper. Ground truth is per-marker corners and ids only (no poses, no intrinsics), so the CLI reports quad recall and, with the matching dictionary, id-aware decode recall/precision. No pose numbers are produced, so the Fast/Accurate pose-mode rule does not apply.

# Quad recall (id-agnostic; accepted + decoder-rejected quads within 10 px of a GT centre)
PYTHONPATH=. uv run --group bench tools/cli.py bench real --dataset liu4k

# Adds id-aware decode TP/FP/FN, recall, precision, F1 (markers use ARUCO_MIP_36h12)
PYTHONPATH=. uv run --group bench tools/cli.py bench real --dataset liu4k --family ArUcoMip36h12

# Variant: sharpening off (the LocalMean (opt-in) threshold variants are analysed in the real-image snapshot)
PYTHONPATH=. uv run --group bench tools/cli.py bench real --dataset liu4k --family ArUcoMip36h12 --no-sharpening

The scorer mirrors aruco_nano's testperf.cpp: for each detection, the first unmatched GT marker with the same id and a centre distance <= 10 px makes it a TP, otherwise it is a FP; FN = GT - TP. The 10 px radius is the Liu4K-specific LIU4K_MATCH_THRESHOLD_PX (tools/bench/liu4k.py); the repo-wide MATCH_DISTANCE_THRESHOLD_PX is untouched. Results and analysis: real-image competitiveness snapshot.

Corner error. bench real --dataset liu4k does not report it. cargo xtask sota run liu4k does: detections are paired with GT markers by id, the GT corner winding (opposite to Locus/OpenCV) is remapped by one fixed relabelling, the error stays order-preserving, and the judged cell is the debiased median on tags every detector found (each detector's mean radial offset is removed and reported separately). Current numbers: 2026-10-04 scoreboard.

cargo xtask data fetch liu4k downloads liu4k.zip (about 4 GB) from Zenodo into tests/data/liu4k/ (gitignored), verifies its md5 (e8fafe5444a9e346f25123151ef1a699, from the Zenodo record), extracts it and deletes the archive; the bench CLI fetches it the same way on first use. The data is never committed, never packaged in wheels or sdist, and never republished in converted form.

Attribution and license. Dataset: Muñoz-Salinas, R., Liu4K dataset employed for Aruco_Nano paper, Zenodo, 2026, doi:10.5281/zenodo.18667018, licensed CC BY 4.0. Locus reads the files unmodified; any published result derived from them must credit the author, cite the DOI, link the license and indicate changes. The record's free-text copyright field also reads "CC0 License"; we apply the CC BY 4.0 attribution terms regardless. Funded by PID2023-147296NB-I00 (Ministerio de Ciencia, Innovacion y Universidades). The dataset's authors also ask users of aruco_nano to cite: R. Muñoz-Salinas, F. J. Romero-Ramirez, S. Garrido-Jurado, "ArUco Nano: a simpler, faster, and more reliable fiducial marker detector" (2026); S. Garrido-Jurado et al., "Automatic generation and detection of highly reliable fiducial markers under occlusion", Pattern Recognition, 2014; F. J. Romero-Ramirez et al., "Speeded up detection of squared fiducial markers", Image and Vision Computing, 2018. Any benchmark report on this dataset must also carry lscpu hardware metadata, build profile, thread count and env vars (docs/engineering/constraints.md section 6).

Hub Dataset Evaluation

Evaluate against rendered Hugging Face Hub datasets. These datasets include ground-truth 6-DOF poses, so the CLI reports both recall and pose error (translation RMSE in metres).

[!NOTE] Pose convention: Hub ground truth poses use a center origin (the pose describes the tag center). Locus reports poses at the top-left corner origin. The CLI automatically applies the rigid center-to-top-left shift via Metrics.align_pose before computing the error.

# Single-tag evaluation
PYTHONPATH=. uv run --group bench tools/cli.py bench real \
  --hub-config locus_v1_tag36h11_1920x1080

# Board-level evaluation (AprilGrid or ChAruco)
# The board topology is inferred automatically from the dataset's rich_truth.json.
PYTHONPATH=. uv run --group bench tools/cli.py bench real \
  --hub-config aprilgrid_golden_v1_1920x1080

PYTHONPATH=. uv run --group bench tools/cli.py bench real \
  --hub-config charuco_golden_v1_1920x1080

# Limit frames and use a custom cache directory
PYTHONPATH=. uv run --group bench tools/cli.py bench real \
  --hub-config aprilgrid_golden_v1_1920x1080 \
  --data-dir tests/data/hub_cache \
  --limit 50

Hub evaluation is mutually exclusive with ICRA scenarios — passing --hub-config skips the --scenarios loop.

Regression Tracking (Baselines)

You can save a "Golden Baseline" and compare current performance against it.

# Save a baseline
PYTHONPATH=. uv run --group bench tools/cli.py bench real --scenarios forward --save-baseline docs/engineering/benchmarking/baseline.json

# Compare current run against baseline
PYTHONPATH=. uv run --group bench tools/cli.py bench real --scenarios forward --baseline docs/engineering/benchmarking/baseline.json

Deep Profiling (Tracy)

Locus supports high-fidelity profiling using the Tracy Profiler.

  1. Rebuild with Tracy support: 
    uv run maturin develop -r -F tracy
  2. Start the Tracy GUI client.
  3. Run benchmark with profiling flag: 
    # Add --profile to any 'bench real' command
PYTHONPATH=. uv run --group bench tools/cli.py bench real --profile --limit 5
    Note: On some Linux systems, you may need TRACY_NO_INVARIANT_CHECK=1 if your CPU doesn't support invariant TSC.

Visual Debugging with Rerun

For diagnosing recall issues or tuning parameters:

uv run tools/cli.py visualize --scenario forward --limit 5

Features

  • Convergence Tracking: Visualize subpixel jitter (yellow arrows) and reprojection errors (scalar plots) for every tag.
  • Failure Diagnosis: Differentiate between geometric rejections (Red) and decoding failures (Orange).
  • Remote & Edge Ready: Debug edge devices remotely using --rerun-addr to stream to a local Rerun viewer.

For a comprehensive walkthrough, see the How-to Guide: Debugging with Rerun.