vernier.panoptic
Panoptic-quality (PQ) evaluation. Evaluator + evaluate(...) mirror
the instance API; Predictions / Dataset carry the panoptic-specific
payload (segment id maps + segment-info JSON).
Panoptic-quality (PQ) evaluation surface (ADR-0025).
Per ADR-0029, the panoptic-segmentation evaluation paradigm lives under
vernier.panoptic. Sibling to :mod:vernier.instance and
:mod:vernier.semantic. The Rust kernel ships in the
vernier-panoptic crate; this module is a thin Python wrapper.
BackgroundEvaluator
BackgroundEvaluator(
categories: bytes,
parity_mode: str,
*,
things_stuff_split: bool = ...,
retain_per_image_deltas: bool = ...,
rank_id: int | None = ...,
queue_capacity: int = ...,
worker_affinity: int | None = ...,
worker_nice: int = ...,
shutdown_timeout_seconds: float = ...,
boundary: bool = ...,
dilation_ratio: float = ...,
num_threads: int | None = ...,
)
Background panoptic-quality evaluator (ADR-0014 + ADR-0032).
Wraps a single dedicated worker thread that owns the underlying
[StreamingPanopticEvaluator]. submit(image_id, gt_label_map,
gt_segments_info, dt_label_map, dt_segments_info) posts one image;
snapshot / finalize block on a worker reply. Mirrors the
instance and semantic background surfaces.
retain_per_image_deltas=True opts into the strict-mode bit-
equality property at merge time (ADR-0032 §"Determinism") at the
cost of ~2× streaming memory, same as the sibling streaming
evaluator.
finalize
method descriptor
finalize() -> PanopticSummary
Drain the queue, finalize the evaluator, and join the worker.
finalize_to_partial
method descriptor
ADR-0032 / ADR-0035: drain, serialize the final state, and shut the worker down.
submit
method descriptor
submit(
image_id: int,
gt_label_map: NDArray[uint32],
gt_segments_info: bytes,
dt_label_map: NDArray[uint32],
dt_segments_info: bytes,
*,
timeout: float | None = None,
) -> None
Submit one image's GT/DT pair to the worker. timeout mirrors
the instance / semantic background surfaces.
submit_png
method descriptor
submit_png(
image_id: int,
gt_png_bytes: bytes,
gt_segments_info: bytes,
dt_png_bytes: bytes,
dt_segments_info: bytes,
*,
timeout: float | None = None,
) -> None
Variant of [Self::submit] that takes panoptic PNG byte blobs
instead of pre-decoded uint32 ndarrays. Fuses libpng decode +
RGB→id + (DT side) S3 area marginals + S1/S11 validation in
one Rust pass; skips the Pillow → numpy → uint32 round-trip
the Python wrapper would otherwise drive on the main thread.
Strictly equivalent to
submit(decode_label_map_png(p), ...) on the result; the diff
is wall-time, not correctness.
Breakdown
Python wrapper around [Breakdown] / [ClassGroupBreakdown].
buckets
property
Range buckets as a list of (label, lo, hi) triples in
construction order.
Raises AttributeError if this Breakdown was built via
from_class_groups. Use class_groups instead.
class_groups
property
Class-id groups as a list of (label, class_ids) pairs in
construction order.
Raises AttributeError if this Breakdown was built via
from_ranges. Use buckets instead.
kind
property
Variant discriminator: "range" for from_ranges-constructed
breakdowns, "class_groups" for from_class_groups-constructed
ones. Use this to dispatch in validators that accept a
Breakdown of a specific shape.
from_class_groups
builtin
from_class_groups(
axis: str, groups: Sequence[tuple[str, Sequence[int]]]
) -> Breakdown
Construct from class-id-keyed groups.
groups is a sequence of (label, class_ids) pairs, one per
group. Group order on input determines the group axis index
(first pair is index 0). Strict partition discipline is enforced
— no class id may appear in two groups.
Raises ValueError on:
- empty
groups; - any group with empty
class_ids; - duplicate group labels;
- the same class id appearing in more than one group.
from_ranges
builtin
from_ranges(
axis: str, buckets: Sequence[tuple[str, float, float]]
) -> Breakdown
Construct from f64-keyed buckets.
buckets is a sequence of (label, lo, hi) triples, one per
bucket. [lo, hi] is closed on both ends per ADR-0016 (quirk
D6); an annotation whose key sits exactly on a boundary lands in
both adjacent buckets.
Raises ValueError on:
- empty
buckets; - NaN or infinite
lo/hi; lo < 0;lo > hi;- duplicate bucket labels.
ClassPanopticStats
PartialDatasetMismatch
Bases: builtins.RuntimeError
Distributed-eval partial was computed against a different dataset than the receiving evaluator.
Attributes: expected (bytes), actual (bytes).
PartialFormatMismatch
Bases: builtins.RuntimeError
Distributed-eval partial blob is structurally malformed (magic / version / CRC / kernel kind / parity / retain_iou / grid dimensions / rkyv archive).
Attributes: kind (string discriminator).
PartialParamsMismatch
Bases: builtins.RuntimeError
Distributed-eval partial was computed against different evaluation params than the receiving evaluator.
Attributes: expected (bytes), actual (bytes).
PartialPartitionOverlap
Bases: builtins.RuntimeError
Two distributed-eval partials cover the same image_id (sampler bug).
Attributes: rank_a, rank_b, image_id.
PartialRankCollision
Bases: builtins.RuntimeError
Two strict-mode distributed-eval partials share a rank_id.
Attributes: rank_id.
Dataset
Parsed-once panoptic ground-truth handle. Build via
[PyPanopticDataset::from_arrays]; pass to
[evaluate_panoptic].
from_arrays
staticmethod
from_arrays(
label_maps: dict[int, NDArray[uint32]],
segments_info: bytes,
categories: bytes,
) -> PanopticDataset
Build a dataset from pre-decoded uint32 label maps. label_maps
maps image id (int) to a 2-D numpy.ndarray of dtype uint32
whose pixel values are panoptic segment ids
(id = R + 256*G + 256²*B, post-rgb2id). segments_info and
categories are JSON byte strings.
image_ids
method descriptor
Image ids present in this dataset. Order is unspecified — the caller must sort if a deterministic order is required.
subset_by_image_ids
method descriptor
subset_by_image_ids(ids: list[int]) -> PanopticDataset
Build a new PanopticDataset retaining only the entries whose
image id is in ids (ADR-0046 §"Panoptic / semantic partitioning").
Ids absent from this dataset are skipped silently — the Python-level partition orchestrator has already done the known-key intersection at manifest parse time. Categories are preserved verbatim (they are a dataset-level taxonomy, not per-image).
Predictions
Parsed-once panoptic prediction handle. Sibling shape to
[PyPanopticDataset]; predictions never carry a category taxonomy
(quirk S9).
num_segments
property
Total number of DT segments across all images. Used as the
n_detections cell on the panoptic slices table (ADR-0046).
from_arrays
staticmethod
from_arrays(
label_maps: dict[int, NDArray[uint32]],
segments_info: bytes,
) -> PanopticPredictions
Build predictions from pre-decoded uint32 label maps. Mirrors
[PyPanopticDataset::from_arrays] but takes no categories.
The DT-side validation runs the full S1/S11 PNG-vs-segments_info
cross-check, recomputes per-segment areas from PNG marginals
(quirk S3), and rejects duplicate ids
([PanopticError::DuplicateSegmentId]).
image_ids
method descriptor
Image ids present in this predictions handle. Order is unspecified — the caller must sort if a deterministic order is required.
num_segments_for
method descriptor
Total number of DT segments across the images in ids. Lets
the Python-side partition loop count n_detections per slice
without rebuilding the per-slice [PanopticPredictions] just
to read its [Self::num_segments].
subset_by_image_ids
method descriptor
subset_by_image_ids(ids: list[int]) -> PanopticPredictions
Build a new PanopticPredictions retaining only the entries
whose image id is in ids (ADR-0046 §"Panoptic / semantic
partitioning"). Mirrors
[PyPanopticDataset::subset_by_image_ids]; predictions do not
carry a category taxonomy (quirk S9), so nothing else
needs to be threaded through.
Summary
Top-level panoptic evaluation result. Read via field accessors.
per_class
method descriptor
per_class() -> dict[int, ClassPanopticStats]
Per-class rows keyed by category id. Returns a Python dict
(constructed fresh on each call from the underlying BTreeMap).
per_group
method descriptor
Per-group rollup keyed by group label (ADR-0042). Empty when
the evaluator was run without class_grouping. Returns a fresh
dict on each call.
to_dict
method descriptor
dict[str, dict | None] matching panopticapi's pq_compute
return shape exactly when strict=True (the count fields are
dropped from per_class to match W8). Useful for round-tripping
with downstream tools that expect the upstream dict layout.
CategoryFilterByGrouping
dataclass
Match every class id in the named group of the active
class_grouping breakdown.
Only meaningful when the Evaluator's class_grouping is also
set; the validator at __post_init__ rejects ByGrouping
when no grouping is configured or when label is not a
grouping label.
CategoryFilterByIds
dataclass
Match an explicit set of class / category ids.
CategoryFilterFrequency
dataclass
Match by LVIS frequency tag ("r", "c", "f").
Valid only on instance evaluation against an LVIS-shaped dataset (ADR-0026). Semantic and panoptic Evaluators reject this variant at construction time per ADR-0041 / ADR-0042 — frequency tags are a sum type that doesn't generalize to non-numeric axes; class groupings carry the user's per-group rollup intent on those paradigms.
InvalidEvalParams
Bases: ValueError
Base for paradigm-specific Evaluator construction errors.
Raised at Evaluator.__post_init__ time by every paradigm in
response to invalid parameter values (out of range, wrong shape,
duplicate, conflicting, etc.). Per ADR-0039, validation runs at
construction so misconfiguration surfaces fast — evaluate()
cannot fail on misconfigured params, only on bad data.
Each subclass carries the offending field name, the offending value, and a one-line remediation pointer (typically the relevant ADR or doc page).
InvalidPanopticParams
EvalResult
dataclass
EvalResult(
summary: PanopticSummary,
_per_class_batch: object | None = None,
_slices_batch: object | None = None,
overall_n_images: int | None = None,
overall_n_detections: int | None = None,
)
Opt-in result of :meth:Evaluator.evaluate when tables= is
passed. Carries :class:Summary plus a polars DataFrame view of
the per-class panoptic-quality breakdown.
Also returned when manifest= is passed (ADR-0046 partitioned
eval): the .summary field carries the bit-identical-to-
un-partitioned overall summary, and the :attr:slices
DataFrame view exposes the per-(axis, value) cell metrics.
per_class
cached
property
One row per category. Columns: category_id, pq,
sq, rq, n_tp, n_fp, n_fn, iou_sum.
StuffThingPartition
dataclass
User-supplied override of the GT-derived stuff/thing split (ADR-0042).
Setting this on :class:Evaluator.stuff_thing_partition overrides
the dataset's isthing flag per category for the purpose of the
PQ_St / PQ_Th rollup. stuff and things must be disjoint
and both non-empty (validated at construction). Membership against
the dataset's category set is checked at evaluate() time once
the Dataset is in scope.
Evaluator
dataclass
Evaluator(
parity_mode: ParityMode = "corrected",
things_stuff_split: bool = True,
boundary: bool = False,
dilation_ratio: float = 0.02,
pq_iou_threshold: float | None = None,
category_filter: CategoryFilter | None = None,
class_grouping: Breakdown | None = None,
stuff_thing_partition: StuffThingPartition
| None = None,
)
Panoptic-quality (PQ) evaluator (ADR-0025, ADR-0042).
Sibling to :class:vernier.instance.Evaluator. Per category,
PQ_c is computed directly as
iou_c / (TP_c + 0.5*FP_c + 0.5*FN_c) (panopticapi form, quirk
W1) — algebraically equal to SQ_c * RQ_c but f64
non-associative, so the direct form is what holds bit-equality.
Global PQ / SQ / RQ are unweighted means over the
present-categories subset (W2, W3, W7); things / stuff buckets are
independent unweighted means over their subsets (W4).
Defaults match panopticapi's pq_compute shape, except for
parity_mode, which defaults to "corrected" (the ADR-0002
recommendation for net-new users); migrating users wanting
bit-exact panopticapi behavior should set parity_mode="strict".
boundary=True enables Boundary Panoptic Quality (ADR-0025 Z1/Z2
amendment). The per-image matching IoU is composed as
min(mask_iou, boundary_iou) — identical to the instance
Boundary case — over a boundary panoptic map built by eroding each
segment's mask by dilation_ratio * sqrt(h² + w²) Chebyshev
pixels and painting the resulting band back as the segment id.
dilation_ratio defaults to 0.02 (Cheng et al. 2021).
FN/FP attribution is unchanged: it consults the mask confusion
matrix and mask per-segment areas only.
The four ADR-0042 fields parameterize the evaluation scope and
rollup: pq_iou_threshold overrides the canonical 0.5 PQ match
threshold (single float, not a ladder); category_filter and
class_grouping mirror the semantic surface (ADR-0041);
stuff_thing_partition overrides the dataset-derived
stuff/thing split for the PQ_St / PQ_Th rollup.
PR scope cut: kernel-side plumbing for honoring the four
custom fields (per_group rollups, threshold-aware matching,
partition override) lands alongside the ADR-0039 distributed-eval
phase. Until then, evaluate() raises
:class:NotImplementedError when any custom field is set; the
surface — fields, validation, StuffThingPartition value type —
is in place.
evaluate
evaluate(
gt: PanopticDataset,
dt: PanopticPredictions,
*,
tables: None = None,
manifest: None = None,
cross_axes: None = None,
num_threads: int | None = None,
retain_per_image_deltas: bool = False,
) -> PanopticSummary
evaluate(
gt: PanopticDataset,
dt: PanopticPredictions,
*,
tables: Literal["all"] | tuple[TableName, ...],
manifest: None = None,
cross_axes: None = None,
num_threads: int | None = None,
retain_per_image_deltas: bool = False,
) -> EvalResult
evaluate(
gt: PanopticDataset,
dt: PanopticPredictions,
*,
tables: None = None,
manifest: object,
cross_axes: Sequence[Sequence[str]] | None = None,
num_threads: int | None = None,
retain_per_image_deltas: bool = False,
) -> EvalResult
evaluate(
gt: PanopticDataset,
dt: PanopticPredictions,
*,
tables: Literal["all"]
| tuple[TableName, ...]
| None = None,
manifest: object | None = None,
cross_axes: Sequence[Sequence[str]] | None = None,
num_threads: int | None = None,
retain_per_image_deltas: bool = False,
) -> PanopticSummary | EvalResult
Run the panoptic-quality evaluation.
gt and dt must have been built via
:meth:Dataset.from_arrays / :meth:Predictions.from_arrays
(file-loading helpers ship in a follow-up).
tables= is the opt-in keyword for result tables (ADR-0038).
Defaults to None, returning :class:Summary (existing
behavior, bit-identical to the pre-tables release). Pass
"all" or a tuple of :data:TableName values to opt into
the wider :class:EvalResult return type.
manifest= opts into ADR-0046 partitioned eval. Accepts a
dict (the canonical JSON-records shape), a file path
(.json), or any object exposing the Arrow PyCapsule
Interface (a polars / pandas / pyarrow DataFrame of per-image
metadata). Returns an :class:EvalResult whose .summary
is bit-identical to the un-partitioned call and whose
.slices property is a polars DataFrame with one row per
(axis, value) cell. cross_axes= opts joint cells in
(per ADR-0046 §E2; marginals are the default).
Per ADR-0046's C3 axiom, the matching + attribution pass
runs exactly once regardless of slice count: the panoptic
kernel emits per-image per-category accumulator deltas, and
the partition orchestrator folds + summarizes those deltas
under (a) no filter for overall and (b) each slice's
image-id set for the per-slice rows — the image-axis analogue
of ADR-0026's K-axis subset-at-summarize-time. The
overall summary is bit-identical to the un-partitioned
call by construction (the un-filtered fold reproduces the
canonical aggregation step verbatim).
evaluate_to_partial
evaluate_to_partial(
images: Iterable[
tuple[
int,
NDArray[uint32],
bytes,
NDArray[uint32],
bytes,
]
],
*,
categories: bytes,
rank_id: int,
retain_per_image_deltas: bool = False,
num_threads: int | None = None,
) -> bytes
Run the panoptic evaluation as a per-rank streaming submit and return the serialized partial bytes (ADR-0032, ADR-0035).
images is an iterable of per-image tuples of the form
(image_id, gt_label_map, gt_segments_info, dt_label_map,
dt_segments_info) — the same shape the streaming substrate's
update consumes. The asymmetry with :meth:evaluate
(which takes pre-built :class:Dataset / :class:Predictions)
is intentional: PanopticDataset does not yet expose
per-image accessors, so the streaming path consumes per-image
records directly. A future ADR may close the gap by adding
:class:Dataset accessors.
rank_id identifies this evaluator's rank in a multi-process
eval. retain_per_image_deltas=True is required on every
rank for strict-mode bit-equality across the merge (ADR-0032
§"Determinism") at ~2x streaming memory cost.
The partial bytes can be gathered across ranks and merged on
the head rank with :meth:from_partials to produce a global
:class:Summary.
Per ADR-0042, raises :class:InvalidPanopticParams when any
of pq_iou_threshold / category_filter / class_grouping
/ stuff_thing_partition is set: extending the ADR-0032
wire format to carry the resolved custom axes is a follow-up.
Single-rank custom-params eval works today via :meth:evaluate.
from_partials
classmethod
from_partials(
categories: bytes,
partials: Sequence[bytes],
/,
*,
parity_mode: ParityMode = "corrected",
things_stuff_split: bool = True,
retain_per_image_deltas: bool = False,
boundary: bool = False,
dilation_ratio: float = 0.02,
) -> PanopticSummary
Merge partials (one per rank) into a global :class:Summary
(ADR-0032, ADR-0035).
categories, parity_mode, things_stuff_split,
retain_per_image_deltas, boundary, and dilation_ratio
must match what each rank used to produce its partial.
Mismatches raise the structured Partial* errors re-exported
on this module — boundary-PQ partials carry the dilation ratio
in params_hash so silent boundary/instance mixing is
rejected at envelope-validation time.
background
background(
categories: bytes,
*,
retain_per_image_deltas: bool = False,
rank_id: int | None = None,
queue_capacity: int = 8,
worker_affinity: int | None = None,
worker_nice: int = 5,
shutdown_timeout_seconds: float = 5.0,
num_threads: int | None = None,
) -> BackgroundPanopticEvaluator
Build a :class:BackgroundEvaluator (ADR-0014 + ADR-0032)
that shares this evaluator's parity_mode and
things_stuff_split.
The returned wrapper owns a single dedicated worker thread
running a :class:StreamingEvaluator of the same shape;
:meth:BackgroundEvaluator.submit enqueues per-image
(gt_label_map, gt_segments_info, dt_label_map,
dt_segments_info) tuples and returns immediately. Use this
when the panoptic-quality kernel measurably stalls the
training loop — the per-image attribute pass is the dominant
cost at COCO-panoptic scale.
retain_per_image_deltas=True enables strict-mode bit-
equality across distributed-eval ranks (ADR-0032
§"Determinism") at ~2x streaming memory cost. The five
queueing / scheduling knobs mirror
:class:vernier.instance.Evaluator.background.
num_threads (ADR-0047) is forwarded into the per-worker
rayon pool. The worker drain-batches pending submissions and
applies them via the parallel kernel; single-threaded callers
(num_threads=None or 1) hit the unchanged single-image
path with no batching overhead.
optimal_lrp
optimal_lrp(
gt: PanopticDataset,
dt: PanopticPredictions,
*,
tp_threshold: float | None = None,
tau_grid: object = None,
parity_mode: ParityMode = "corrected",
) -> object
Panoptic LRP / oLRP (ADR-0043 — namespace reserved, not yet wired).
Per ADR-0043 panoptic LRP runs a parallel matching pass that does
not reuse PQ's hard IoU > 0.5 cutoff (the tau sweep needs the
full unthresholded matched-IoU list). That pass is not yet wired
in 0.5.x because the panoptic prediction data model
(vernier.panoptic.Predictions) does not carry per-segment
scores — without a confidence value per detection there is no
tau to sweep, and adding scores requires extending the panoptic
paradigm's wire format (a separate ADR-level decision).
The Python entry point is registered here so from
vernier.panoptic import optimal_lrp works as advertised by
ADR-0043; calling it raises :class:NotImplementedError with a
structured remediation pointer to the same ADR. When the
panoptic prediction shape adds a score field, this stub
delegates to a sibling Rust kernel pass that mirrors the
matching logic in :func:vernier.instance.optimal_lrp over
segment-IoU.
The argument shape is fixed in 0.5.x even though the function raises today: callers can write the call site against the final surface and the upgrade is purely backend.
decode_label_map_png
Decode a panoptic RGB PNG into a (H, W) uint32 segment-id
label map via the rgb2id convention (panopticapi/evaluation.py
rgb2id: r + 256*g + 256²*b).
Lazy-imports Pillow; raises a structured :class:ImportError if it
isn't installed. Three channels are required — a non-RGB PNG is
rejected with :class:ValueError. Single-channel class-id label
maps (semantic-segmentation) belong in
:func:vernier.semantic.Dataset.from_files, which has its own
decoder.