Tutorial: benchmark speed and geometric accuracy

Field

Contract

Application

Evaluate any PanorAi workflow

Input

Frozen source, installed artifact, environment, and public data manifest

Output

Reproducible correctness, runtime, memory, and limitation evidence

Strategy

Cross-cutting evaluation

API

Benchmark scripts and public result objects

Stability

Guidance; evaluated API keeps its own level

Dependencies

Defined by the evaluated workflow

Evidence

Protocol, raw measurements, and artifact identity

How this application is composed

Benchmarking does not add a numerical operator. It executes one public core operation or application wrapper against a pinned input, records warm-up, repetitions, environment, median, P95 and memory, and keeps raw measurements. This separation lets a wrapper benchmark expose both end-to-end cost and the core stages responsible for it.

This tutorial turns a PanorAi experiment into evidence that someone else can interpret and reproduce. It distinguishes microbenchmarks, stage metrics, and end-to-end success; none can substitute for the others.

1. Freeze the evaluation unit

The independent unit is a physical scene group, not an image, crop, face, patch, pair, or multiview set.

  • keep every Matterport3D panorama from one building in one split;

  • keep every Stanford2D3D panorama from one area in one split;

  • freeze selection rules before reading test outcomes;

  • report Matterport3D and Stanford2D3D separately before any aggregate;

  • record whether ground truth was available to selection, tuning, or only final scoring.

PanorAi does not distribute either dataset. Follow the dataset owners’ licenses and point benchmark manifests at your local copies.

2. Benchmark the installed artifact

Running inside a checkout can import local source instead of the built wheel. Create a clean environment, install the wheel under test, change to a directory outside the repository, and record:

  • panorai.__version__ and panorai.__file__;

  • Python, NumPy, OpenCV, OS, architecture, and CPU/GPU;

  • input shape, dtype, layout, FOV, face resolution, interpolation, and blender;

  • first-call time separately from reused calls;

  • warm-up count, repetitions, median, P95, and peak memory when relevant.

For the two image-processing routes:

python /path/to/PanorAi/scripts/benchmark_image_processing_routes.py \
  --require-installed \
  --backend native \
  --output image-processing-routes.json

The runner starts each route in a fresh process. It times direct spherical Gaussian filtering and the complete project/filter/reconstruct route, then evaluates both with the same yaw-equivariance probe.

3. Measure each stage, then the full pipeline

For detection and relative pose, store at least:

Stage

Runtime

Accuracy / quality

preprocessing

wall time and peak memory

dynamic range, valid support

detection

pyramid, extrema, localization, selection

keypoint count, angular coverage, repeatability

description

patch materialization and descriptor compute

valid descriptor rate, duplicate/ambiguity rate

matching

search and filtering

match count, precision against GT epipolar compatibility, spatial coverage

robust pose

hypotheses, scoring, refit, quality gate

inlier precision/recall, residual, accepted/rejected reason

end to end

total cold and reused time

rotation error, translation-direction error, success and abstention

Do not infer that a faster detector gives better pose, or that more matches give better pose. Preserve intermediate artifacts so a failure can be assigned to detection, description, matching, robust consensus, decomposition, or the acceptance gate.

4. Report selective estimators honestly

A quality gate creates three different quantities:

  1. coverage — fraction of cases for which a result is accepted;

  2. selective precision — fraction of accepted results that satisfy the declared accuracy criterion;

  3. recall — fraction of target-solvable/correct cases recovered.

Always report them together. “99% precision” without coverage can describe a system that almost always abstains.

For continuous errors, report distributions—not only means:

  • rotation geodesic error in degrees;

  • translation-axis and oriented-direction error in degrees;

  • spherical epipolar residual in degrees;

  • radial-range AbsRel and threshold accuracies;

  • median, P90/P95, and clearly declared failure handling.

5. Separate tuning from evidence

Use development groups to choose patch FOV, descriptor resolution, Lowe ratio, RANSAC thresholds, and gate parameters. The frozen test split may be scored once for the registered configuration; it must not guide a second choice.

Every result record should contain:

  • exact command and resolved configuration;

  • generating commit and installed version;

  • dataset, split, group IDs, and oracle-access declaration;

  • coordinate frame, units, pose direction, and scale convention;

  • counts before and after every filter/gate;

  • machine-readable metrics plus a short limitations section;

  • checksums or stable URIs for large external artifacts.

6. Read the current evidence

The project-wide Benchmarks and performance page records the current public-dataset accuracy and installed-artifact performance evidence, including limitations and outstanding promotion gates. Continue with: