Stratification Schema

Purpose. Define how regression metrics are sliced along physically meaningful axes so we catch regressions that averaging hides.

Status. Axes are frozen for v1. Bucket boundaries are locked (see §5); they were derived by tools/bench/strata_histogram.py running 33/66-percentile equal-frequency snapping over the four single_tag_locus_v1_tag36h11_* corpora.

1. Motivation

The baseline (benchmarking/baseline.json) reports one recall number per dataset per library — a 1% drop could be a 4% drop in the 4K stratum masked by 720p gains, a systematic oblique-view loss masked by better frontal detection, or a silent break on distant small tags. Splitting by PPM/AOI/distance/motion targets the physical regime the detector runs in, not the rendering configuration, giving: early-warning (a stratum-scoped gate fires on a 5% dist=far regression even when the global number holds), debuggability (stratum_id narrows the search to a handful of images), and cross-dataset comparability (a stratum means the same thing in every hub-config and in ICRA 2020).

2. The five axes

Every ground-truth record is assigned a bucket on each axis at load time. Source fields come from tests/data/hub_cache/*/rich_truth.json; see tools/bench/utils.py for the existing loader wired to the TagGroundTruth / DatasetMetadata dataclasses.

Axis	Source field(s)	Unit	Bucket count	Notes
Resolution	resolution: [W, H]	pixels	4 (TBD)	Bucket on H (height). Expected slugs: sd, hd, fhd, uhd.
PPM (pixels per metre of tag)	ppm (preferred) else derived from corners + tag_size_mm	px/m	3 (TBD)	ppm == 0.0 in current rich_truth.json files — see §3. Slugs: lo, mid, hi.
Angle of incidence (AOI)	angle_of_incidence	degrees	3 (TBD)	0° = frontal. Slugs: frontal, oblique, grazing.
Distance	distance	metres	3 (TBD)	Slugs: near, mid, far.
Motion	velocity or shutter_time_ms × ‖velocity‖	m/s (or px blur)	2 (TBD)	Most current records have velocity: null ⇒ static. Slugs: static, motion.

Axis selection rationale

• Resolution, PPM, AOI, distance are the canonical four of fiducial-detector evaluation literature. They are present, per-tag, in the existing ground truth.
• Motion is nascent. Today most records are velocity: null and rolling_shutter_ms: 0.0. Rather than drop it and rebuild the schema later, v1 reserves the slot with a degenerate static bucket; synthetic datasets or future captures populate the motion bucket.

3. Known data-quality gaps

These are called out so reviewers don't discover them mid-triage:

• ppm is 0.0 in the current hub-cache rich_truth.json files. The axis is still load-bearing — the loader must derive PPM from: ppm ≈ max_edge_px(corners) / (tag_size_mm / 1000.0). Boundaries for the ppm axis should be chosen on the derived value, not the raw field.
• velocity: null for all current hub-cache records. Treated as static. When a dataset with motion lands, the loader reads velocity directly (or computes effective blur as shutter_time_ms * ‖velocity‖ * ppm).
• ICRA 2020 has no per-tag metadata; every record there collapses to res=<from image shape>|ppm=unk|aoi=unk|dist=unk|mot=unk. The schema must accept unk for any axis without crashing the loader.

4. Canonical stratum_id format

stratum_id is a single string, deterministic across runs, diffable in baseline_v2.json entries.

Grammar

stratum_id := axis_pair ("|" axis_pair){4}
axis_pair  := key "=" bucket
key        := "res" | "ppm" | "aoi" | "dist" | "mot"
bucket     := slug | "unk"
slug       := [a-z0-9]+          # short, lowercase, alphanumeric

Rules

1. Key order is fixed: res, ppm, aoi, dist, mot. Always five pairs, always in that order, always |-separated.
2. Slugs are short and lowercase. No numeric ranges inside the id — ranges live in the boundary table (see §5). This keeps ids stable across re-bucketing.
3. Unknown bucket is unk. Any axis that cannot be derived from a record (missing source field, NaN, out-of-range) uses unk. Reporters treat unk as its own legitimate stratum — it is visible in diffs and has its own tolerances.
4. Escape rules. Slugs must not contain = or |. The validator in tools/bench/schema.py:Tolerances (A0.2) rejects ids that fail the grammar.

Examples

res=fhd|ppm=hi|aoi=frontal|dist=near|mot=static
res=uhd|ppm=lo|aoi=grazing|dist=far|mot=static
res=hd|ppm=unk|aoi=unk|dist=unk|mot=unk       # ICRA 2020 record

5. Bucket boundary table

Derivation: tools/bench/strata_histogram.py over the pooled single_tag_locus_v1_tag36h11_* corpora (n=200), 33/66 percentiles snapped to a human-readable grid (dist→0.1m, aoi→5°, ppm→100). The snap audit caps reshuffle versus raw percentiles at 15% of the pool; current snap moves 19/200 records (≤2.5pp per axis from equal-frequency).

The constants below are mirrored in tools/bench/strata.py. Editing this table is a v1 patch bump (see §6) — re-run strata_histogram.py and update both files in lockstep.

Axis	Slug	Range	n
res	sd	H ≤ 480	50
res	hd	480 < H ≤ 720	50
res	fhd	720 < H ≤ 1080	50
res	uhd	1080 < H	50
ppm	lo	ppm ≤ 800	71
ppm	mid	800 < ppm ≤ 1300	58
ppm	hi	1300 < ppm	71
aoi	frontal	angle ≤ 35°	66
aoi	oblique	35° < angle ≤ 50°	58
aoi	grazing	50° < angle	76
dist	near	d ≤ 0.7	69
dist	mid	0.7 < d ≤ 1.5	63
dist	far	1.5 < d	68
mot	static	‖velocity‖ ≤ 0.0 (or null)	200
mot	motion	0.0 < ‖velocity‖	0

6. Extension protocol

Adding, removing, or renaming an axis is a schema minor bump (v1.x → v1.(x+1)) and requires:

1. Update this doc: add the axis row to §2, update the stratum_id grammar in §4, add boundary entries in §5.
2. Update tools/bench/schema.py validators.
3. Bump BaselineV2.schema_version (see tools/bench/schema.py).
4. Write a migration for existing baselines (old ids need the new axis populated with unk).

Re-bucketing an existing axis (changing cut points only) is a v1 patch bump and does not change the stratum_id grammar — but it does invalidate existing baselines because the same raw record may map to a different bucket. Re-run the baseline after re-bucketing.

7. Sources

• Field schema: tests/data/hub_cache/*/rich_truth.json (sampled).
• Loader types: tools/bench/utils.py — TagGroundTruth, DatasetMetadata, HubDatasetResult.
• Related benchmarking contract: docs/engineering/benchmarking.md.