Semantic object association and localization¶

panorai.object_localization is the Experimental panorai-object-localization/v1 API. Its compatibility path estimates semantic object hypotheses from exactly two views. The additive panorai-multiview-object-hypotheses/v1 path consumes an already registered spherical reconstruction and dense semantic evidence from two or more views. panorai.graph may persist and connect accepted evidence, but this module does not mutate or construct a graph.

Simple v1 flow¶

The caller supplies:

a required SemanticQuery, preserving the original text and the class IDs resolved in a declared vocabulary;
one or more CAM/semantic regions per panorama;
panorai.features.SphericalFeatureSet objects and their panorai.features.SphericalFeatureMatches;
an accepted panorai.estimators.RelativePoseResult; and
optionally, a positive translation scale.

The query defaults to the imagenet-1k vocabulary. V1 deliberately does not parse unrestricted language: a detector, classifier, synonym table, or UI resolves the text to explicit vocabulary IDs and records that decision in the query. Region proposals outside those IDs are ignored.

The pipeline is deliberately small:

textual query + regions/maps + spherical features + matches + relative pose
    -> propose class-compatible region pairs
    -> one-to-one assignment with an ambiguity gate
    -> deterministic ObjectHypothesis ID
    -> SpatialLocationHypothesis

The same pair of (view_id, region_id) observations always produces the same direction-independent object ID. It identifies the object hypothesis shared by those two images. Persistent identity across three or more views, track merging, split/merge resolution, and lifecycle management belong to the later graph stage.

Registered multiview flow¶

SphericalObjectHypothesisMethod is the high-level Experimental facade for the complete implemented methodology. It does not download or run a classifier, depth estimator, feature encoder, or segmenter. The caller supplies those evidence records explicitly:

from panorai.object_localization import SphericalObjectHypothesisMethod

method_result = SphericalObjectHypothesisMethod().run(
    reconstruction=registered_reconstruction,
    semantic_evidence=evidence,
    concepts=concepts,
    radial_depth=registered_radial_depth,  # optional radial range
    independent_region_provider=infer_each_view_region_independently,
)

The result retains generation, supported regional utility, track/reprojection and optional embedding consistency, compact registered-transfer summaries, the exact pair cues used for reduction, and the reduction result. Transfer masks are transient and are not retained.

Automatic reduction transfers are evaluated only between hypotheses with the same exact vocabulary and concept. Incompatible concepts already fail the hard affinity gate, so rasterizing their regions would add cost without creating admissible evidence. All hypothesis pairs still appear in the final auditable affinity table.

Every generated hypothesis exposes a provenance mapping containing the geometry contract, spatial-point source, reconstruction scale, exact concept and proxy weights, semantic evidence sources and their upstream provenance, the frozen generator configuration, and the clustering/state trace. Point-free results additionally retain per-track triangulation quality.

Registered transfer is deliberately disabled unless the caller supplies per-view regions through independent_regions_by_hypothesis or independent_region_provider. The ordinary generated view_regions share one reconstructed support cluster and therefore cannot be used to claim independent transfer agreement. This gate prevents the tautological comparison prohibited by the methodology.

SphericalObjectHypothesisGenerator treats a CAM or similarity map as directional semantic evidence, not as an object mask. Geometry and correspondence identity come from either accepted sparse landmarks in a successful refined panorai.reconstruction.SphericalReconstructionResult, or from known camera poses plus cross-view bearing tracks that contain no input 3-D points. For every active track observation, the generator samples the semantic field at the observation bearing. Tracks with compatible evidence are clustered in the registered world frame and promoted conservatively only after the configured number of independent semantic views and supporting tracks is reached:

from panorai.object_localization import (
    RegisteredSemanticEvidence,
    SemanticConcept,
    SphericalObjectHypothesisGenerator,
)

evidence = tuple(
    RegisteredSemanticEvidence.from_dense(view_id, dense_cam)
    for view_id, dense_cam in dense_cams_by_view.items()
)
concepts = (
    SemanticConcept(
        concept_id="piping",
        display_name="Piping",
        proxy_weights=(("coil", 1.0), ("reel", 1.0)),
        vocabulary="industrial-proxy/v1",
    ),
)
result = SphericalObjectHypothesisGenerator().generate(
    registered_reconstruction,
    evidence,
    concepts,
)
for hypothesis in result.confirmed:
    print(hypothesis.concept_name, hypothesis.spatial.center_xyz)
    for region in hypothesis.view_regions:
        print(region.view_id, region.spec, region.sparse_support_coverage)

If refined sparse landmarks are not available, use the no-point entry point:

import numpy as np
from panorai.object_localization import (
    RegisteredBearingObservation,
    RegisteredBearingTrack,
)
from panorai.reconstruction import SphericalCameraPose

# Given x_v = R_v @ X + t_v, PanorAi's equivalent camera centre is
# C_v = -R_v.T @ t_v and its convention is x_v = R_v @ (X - C_v).
poses = tuple(
    SphericalCameraPose(view_id, R_v, -R_v.T @ t_v)
    for view_id, R_v, t_v in registered_rt
)
tracks = tuple(
    RegisteredBearingTrack(
        track_id,
        tuple(
            RegisteredBearingObservation(view_id, feature_index, bearing)
            for view_id, feature_index, bearing in observations
        ),
    )
    for track_id, observations in matched_bearing_tracks
)
result = SphericalObjectHypothesisGenerator().generate_from_bearings(
    poses=poses,
    tracks=tracks,
    semantic_evidence=evidence,
    concepts=concepts,
    scale="arbitrary",  # use a declared metric scale only when t is metric
)

generate_from_bearings estimates each latent track point by weighted multiray intersection. bearing_track_triangulations reports parallax, normal-matrix conditioning, positive-depth fraction, angular reprojection error, and ray-distance residual for every track. An accepted point_xyz is an estimate produced by this operation, not ground truth.

Low parallax or an ill-conditioned intersection is a local track degeneracy. The track is excluded without invalidating otherwise coherent poses. In contrast, when the configured fraction of non-degenerate tracks fails cheirality or angular reprojection, the generator raises a pose-consistency error. Specifically, RegisteredPoseInconsistencyError exposes the input, non-degenerate, and rejected track counts, rejected fraction, and reason histogram. Callers can therefore record pose_inconsistent without parsing text. The method does not hide systematic disagreement by returning a collection of bearing_only hypotheses. This is the explicit signal to inspect R,t.

Do not pass points assembled from a depth raster, scanner XYZ, or a one-shot reprojection filter to generate() and call them refined landmarks. Either the points and active observations survived a declared geometric-refinement stage, or the caller must discard their coordinates and use the point-free bearing path above.

RegisteredSemanticEvidence is NumPy-only and keeps scores (CHW) separate from a geometric support mask (HW). Its from_dense adapter accepts the Experimental DenseSemanticEvidence contract by attributes, so importing panorai.object_localization does not import Torch, SAM, or the segmentation package.

One canonical concept may list several proxy channels. Missing proxies contribute zero rather than having their weights silently redistributed, and the hypothesis retains every proxy and source identifier as provenance. This supports, for example, a domain concept such as piping backed by explicit ImageNet coil and reel channels without claiming that either class is an industrial ground-truth label.

The returned spatial support is one of:

metric_3d: Weighted reconstructed-track support in the registered world frame when reconstruction declares metric scale. The configured units are explicit.
scale_free_3d: The same finite support in the reconstruction’s arbitrary coordinate scale. It must not be interpreted as metres.
bearing_only: A proposed direction in one panorama when dense evidence exists but no registered semantic track supports it. Bearing-only results are never automatically confirmed or triangulated from unrelated CAM peaks.

extent_covariance_xyz and extent_radius describe the spatial spread of supporting refined or internally triangulated track points. They are not calibrated estimator uncertainty. Per-track triangulation conditioning and residuals remain separate in bearing_track_triangulations. Every finite hypothesis also exposes one HypothesisGnomonicRegion per registered camera. Its centre is the weighted mean direction of the projected sparse support; its rectangular field of view contains that support plus the configured angular margin, up to the configured maximum FOV. The region is a reproducible crop proposal, not an object mask.

hypothesis_view_consistency compares every pair of these view regions. It reports shared registered tracks, feature-index pairs, Jaccard and smaller-view coverage. When the source tracks carry geometric residuals, it also reports the median pair residual and the robust exponential reprojection agreement. Optional caller-owned HypothesisViewEmbedding vectors add cosine similarity without selecting or importing an embedding model. Set preprocessing_id on those embeddings to make the frozen encoder contract auditable; mixed preprocessing IDs fail explicitly. These quantities remain separate rather than being hidden in one uncalibrated confidence score.

Regional utility and registered transfer¶

evaluate_spherical_region() evaluates one independently inferred HypothesisGnomonicRegion against a RegisteredSemanticEvidence. It reports solid-angle-weighted semantic-mass recall, observed-sphere fraction, and mean-score enrichment. Zero semantic mass is valid evidence: recall and enrichment are then None rather than being manufactured with an epsilon. The observed angular fraction remains defined whenever geometric support is non-empty.

RegisteredRadialDepth stores radial range, never camera-axis depth. Its support mask is authoritative and supported values must be finite and strictly positive. One directional transfer is:

from panorai.object_localization import transfer_spherical_region

a_to_b = transfer_spherical_region(
    region_a,
    region_b,              # inferred independently in B
    radial_depth_a,
    pose_a,
    pose_b,
    target_support=radial_depth_b.support,
)
print(a_to_b.solid_angle_iou)

Use symmetric_spherical_region_transfer() when radial depth is available in both views. It transfers A to B and B to A separately and returns their mean solid-angle IoU. The transfer rasterizes boolean membership with nearest pixel-centre assignment and wraps longitude; it never bilinearly interpolates a mask. Independently inferred regions are a precondition. Projecting one common 3-D box into both views and comparing those projections is tautological and is not this metric.

Spherical FOV anchors and multiview consensus¶

panorai-spherical-anchor-fusion/v1 is the detector-style region stage for registered panoramas. A SphericalFOVAnchor is a local gnomonic field-of-view proposal, not an ERP rectangle or an object mask. generate_spherical_fov_anchors() produces deterministic scale/aspect alternatives around one direction.

The one-call SphericalObjectHypothesisMethod converts independently inferred per-view regions into anchors and returns an optional anchor_fusion result. The result applies spherical intra-view NMS, registered pair scoring, deterministic complete-link clustering with at most one anchor per view, cluster-level NMS for duplicate scale/aspect families, and a per-view consensus FOV refit. Suppressed anchor and cluster IDs remain auditable in the result. The method never forces two cameras to use the same angular box: each view retains the extent supported by its local evidence and by transferred support from the other views.

The complete audit trail is typed and mask-free. Every generated anchor has exactly one SphericalAnchorDecision with action retained, suppressed-overlap, or suppressed-capacity. An overlap suppression identifies its representative and solid-angle IoU; a capacity suppression deliberately has neither. Every raw complete-link cluster similarly has one SphericalAnchorClusterDecision with action retained or suppressed-family-overlap. A cluster is suppressed only when it has the same explicit local-component proposal-family IDs as a stronger cluster and its per-view consensus regions meet the configured solid-angle overlap threshold. Sharing an accumulated upstream hypothesis ID is not sufficient: one hypothesis may contain multiple distinct instances or local components. These records preserve compact FOV geometry and scores, never transient ERP masks.

Association cues are typed by SphericalAnchorPairCues. Registered region-transfer IoU, shared-track consistency, and optional appearance cosine similarity are alternative identity evidence. Geometric compatibility is a multiplier, while concept/vocabulary mismatch, same-view alternatives, geometric contradiction, depth separation, and track contradiction are hard gates. None means a cue was not observed; numeric zero means it was observed and found no agreement. These states are never conflated.

Clusters use complete-link acceptance: a new anchor must exceed the frozen affinity threshold against every existing member. This prevents the single-link A--B--C bridge failure in which mutually inconsistent A and C would otherwise be joined through B. A cluster is confirmed only after it contains the configured minimum number of distinct registered views.

Consensus masks are transient. For target view \(v\), the local anchor mask and every available registered transfer into \(v\) are accumulated with the corresponding pair affinities. The thresholded spherical support is then refitted to the smallest configured-coverage gnomonic FOV plus an angular margin. Only the compact region, provenance, and scores are retained; the temporary ERP masks are released.

Three regions therefore have different meanings:

search region: The projection of the current spatial hypothesis used to locate nearby local CAM components. It is not the reported object extent.
local anchor: The FOV fitted from one view’s independently thresholded CAM component.
consensus region: The final per-view FOV after cross-view association and registered support fusion. This is the recommended gnomonic crop for downstream feature or segmentation processing.

Affinity, fusion, and Soft-NMS¶

hypothesis_pair_affinity() keeps the semantic, geometric, track, transfer, and optional appearance factors separate. Unavailable optional cues are omitted from the product; an observed factor equal to zero remains decisive. Vocabulary, concept, spatial-mode, frame, unit, caller-declared geometric, and depth-separation contradictions are hard gates and force zero affinity.

Use reduce_object_hypotheses() with typed HypothesisReductionCandidate records. Edges above affinity_threshold form deterministic duplicate graph components. A finite, mutually hard-compatible component is fused in registered 3-D using semantic_score * geometric_quality * independent_view_factor. The result retains the representative hypothesis, every source hypothesis ID, every original score, and a separate fused_spatial record. It does not pretend that one source view region was re-inferred from the fused centre.

Unfused alternatives are never deleted. Gaussian spherical Soft-NMS decays only their ranking score and records every suppressor ID. Incompatible concepts, frames, units, spatial modes, or explicitly depth-separated supports remain concurrent. The input hypotheses and their semantic scores are immutable.

The deterministic hypothesis ID depends on the canonical concept and track IDs, not input order or a segmentation boundary. Segmentation may refine a confirmed hypothesis later, but it is not required by this workflow.

Spatial meaning¶

SpatialLocationHypothesis has three honest output modes:

metric_3d: A translation scale was supplied. Position and covariance use the caller’s declared units in the first panorama frame.
scale_free_3d: Translation direction is known but scale is not. Position and covariance use a normalized unit baseline in the first panorama frame.
bearing_only: Triangulation did not meet the parallax, positive-depth, support, or reprojection gates. The result retains a unit viewing direction and does not invent a 3D point.

The 3D position is the robust center of triangulated regional feature points; it is not claimed to be the physical center of the object. identity_score and location_score are transparent ranking scores, not calibrated probabilities.

Map adapter¶

semantic_region_from_map() converts one ERP-aligned CAM or semantic score map into a SemanticRegionObservation. It samples the map at existing feature coordinates and stores feature indices rather than copying descriptors. The caller remains responsible for splitting a class activation map into distinct candidate instances.

semantic_regions_from_map() provides the simple built-in alternative when no detector or instance splitter is available. It thresholds one class CAM, labels connected components with periodic horizontal ERP connectivity, and returns deterministic SemanticRegionObservation candidates. Vertical rows do not wrap. Four- and eight-neighbour connectivity are explicit configuration choices.

The accompanying SemanticRegionProposalResult reports raw, retained, too-small, featureless, truncated, and seam-crossing component counts. V1 requires each retained component to contain at least one existing feature. A component is a CAM support island, not a guaranteed object instance: two nearby objects may merge, and one object may fragment into multiple components.

Joint match clusters¶

When one CAM component contains multiple physical objects, cluster_region_association_matches() can refine an already accepted broad region association. Two matches are neighbours only when their bearings are within the configured angular radius in both panoramas. Bearing-space distances cross the ERP longitude seam without a pixel-coordinate special case.

The returned JointMatchClusterResult contains paired feature-indexed subregions. They are ordinary SemanticRegionObservation objects and can be passed through ObjectLocalizationPipeline again:

from dataclasses import replace
from panorai.object_localization import cluster_region_association_matches

clustered = cluster_region_association_matches(
    accepted_broad_association,
    evidence,
)
refined_evidence = replace(
    evidence,
    regions_a=clustered.regions_a,
    regions_b=clustered.regions_b,
)
refined_result = ObjectLocalizationPipeline().estimate(refined_evidence)

By default, only pose inliers participate. Small components are discarded and reported rather than promoted. Cluster IDs depend on the matched feature pairs, not match row order, so repeated execution is deterministic.

This is deliberately a candidate-instance heuristic, not instance segmentation. It uses connected components in a joint angular neighbourhood; dense background bridges can chain two objects together, while a radius that is too small can fragment one object. V1 keeps the radius explicit and does not claim it is calibrated across scenes. On the frozen real office pair, a 1-12 degree sweep created up to 26 localized hypotheses, recovered at most two of three target identities, and never recovered the second repeated desk. Therefore this helper is not enabled automatically by the pipeline and its clusters require an additional instance-aware semantic cue before object-ID promotion.

Query-conditioned pose sampling¶

build_semantic_match_prior() turns those feature-indexed regions into a full-length RANSAC proposal prior. For each class explicitly named by the SemanticQuery, it takes the geometric mean of the feature membership evidence in the two views. Evidence is combined by a maximum, so duplicating a class or region cannot increase its mass.

The returned SemanticMatchPrior has one sampling_weights row per original match. Invalid rows remain zero and every valid row remains strictly positive. Consequently, a CAM can make a minimal set more likely but cannot remove a correspondence from robust scoring or pose refinement. If the query has no shared support, the result is an exact uniform fallback with an explicit fallback_reason:

from panorai.object_localization import build_semantic_match_prior

prior = build_semantic_match_prior(
    query,
    regions_a,
    regions_b,
    matches,
)
correspondences = prior.to_bearing_correspondences(matches)
pose = relative_pose_estimator.estimate(correspondences)

The default convex mixture uses a 0.25 uniform component before normalization and caps the maximum valid-row weight ratio at four. These are conservative uncalibrated defaults, not learned probabilities. A weight ratio of one is exactly the uniform-weight baseline.

Semantic support and geometric suitability are separate. Features on a compact object can have excellent CAM agreement but insufficient angular spread for a conditioned five-point sample. Conversely, a false CAM can concentrate sampling on outliers. Callers must compare the candidate against the ordinary geometric estimate and must not accept it merely because it used semantic evidence or found more inliers.

Minimal composition¶

After a detector or CAM has created the two region observations:

from panorai.object_localization import (
    ObjectLocalizationPipeline,
    PairObjectLocalizationInput,
    SemanticQuery,
)

evidence = PairObjectLocalizationInput(
    view_id_a=features_a.panorama_id,
    view_id_b=features_b.panorama_id,
    query=SemanticQuery(text="find a chair", class_ids=(chair_class_id,)),
    regions_a=(chair_region_a,),
    regions_b=(chair_region_b,),
    features_a=features_a,
    features_b=features_b,
    matches=matches,
    pose=relative_pose,
    translation_scale=None,
)
result = ObjectLocalizationPipeline().estimate(evidence)

for hypothesis in result.hypotheses:
    print(hypothesis.hypothesis_id, hypothesis.location.mode)

By default, a relative pose rejected by its own quality policy fails closed. Ambiguous assignments remain visible in result.associations but are not promoted into confirmed object hypotheses.

Practical validation¶

benchmarks/object_localization contains a reproducible development validation on a real Stanford2D3D office pair. It estimates features, matches, relative pose, object identity, and scale-free spatial hypotheses from RGB and frozen manual regions before opening the reference pose or depth.

For the frozen pair, the pipeline associated two distinct desk instances and one bookcase with precision and recall of 1.0. The three hypothesis IDs were unique and repeated identically. Against registered reference data, relative rotation and translation-direction errors were 0.268 and 0.283 degrees; the maximum evaluation-only metric error of the regional 3D centers was 0.014 m. All identities survived a 16-pixel expansion and contraction of every region. At the 32-pixel stress level, insufficient or overlapping region evidence was rejected or marked ambiguous, with no incorrect identity promoted.

This is evidence for association and geometry after candidate regions exist; it is not evidence for detector or CAM quality, and one room is not enough to claim broad dataset generalization.

run_semantic_prior_limits.py is a separate causal synthetic benchmark. Each baseline/guided pair keeps correspondences, validity, estimator options, and random seed fixed and changes only minimal-set proposal weights. It covers correct diffuse support, correct compact support, mixed evidence, a misleading prior, and absent support. This benchmark maps an operating envelope; it is not evidence that ImageNet CAMs achieve the simulated precision or recall on real panoramas.

Public API¶

Experimental semantic association and spatial localization.

The pairwise API composes maps, spherical features, matches, and relative pose. The multiview API composes dense semantic evidence with either refined sparse reconstruction tracks or registered poses plus bearing-only tracks. Neither path mutates a scene graph.

class panorai.object_localization.BearingTrackTriangulation(track_id, state, view_ids, point_xyz, maximum_pairwise_parallax_deg, normal_condition, positive_depth_fraction, maximum_angular_residual_deg, rms_ray_distance, angular_residuals_deg, reasons=())¶

Auditable estimate of one latent track point from registered rays.

point_xyz is an estimate in the pose world frame, never ground truth. normal_condition and the residual fields are quality diagnostics, not a calibrated covariance model.

class panorai.object_localization.DirectionalRegionTransfer(source_view_id, target_view_id, transferred_mask, target_region_mask, target_support, solid_angle_iou, source_region_pixel_count, source_supported_depth_count, transferred_target_pixel_count)¶: One registered region transfer and its target-domain overlap.

class panorai.object_localization.HypothesisAffinity(hypothesis_id_a, hypothesis_id_b, hard_compatible, hard_reasons, factors, affinity, soft_nms_overlap)¶: Auditable compatibility factors and their multiplicative affinity.

class panorai.object_localization.HypothesisAffinityConfig(semantic_exponent=1.0, geometric_exponent=1.0, track_exponent=1.0, transfer_exponent=1.0, appearance_exponent=1.0, affinity_threshold=0.5, geometric_scale_floor=1e-06, soft_nms_sigma=0.5, soft_nms_min_overlap=0.0, fuse_finite_components=True)¶: Frozen factor exponents and reduction thresholds.

class panorai.object_localization.HypothesisGnomonicRegion(view_id, spec, center_bearing_xyz, angular_radius_deg, support_bearings_xyz, track_feature_indices, sparse_support_coverage, semantic_evidence_available, semantic_supporting, extent_source='projected-sparse-tracks-plus-margin')¶

One auditable gnomonic window for a hypothesis in a registered view.

The window is centred on the weighted mean direction of the projected sparse 3D support. Its field of view contains that support plus the configured angular margin, subject to the configured maximum FOV. This is a crop proposal, not a segmentation mask or an estimator covariance.

class panorai.object_localization.HypothesisPairCues(hypothesis_id_a, hypothesis_id_b, geometric_compatibility=None, transfer_iou=None, appearance_cosine_similarity=None, soft_nms_overlap=None, geometric_contradiction=False, depth_separated=False, provenance=<factory>)¶: Caller-supplied optional evidence for one canonical hypothesis pair.

class panorai.object_localization.HypothesisReductionCandidate(hypothesis, geometric_quality=1.0, independent_view_factor=1.0)¶: One hypothesis and the independent factors used for fusion weight.

class panorai.object_localization.HypothesisReductionResult(records: 'tuple[ReducedHypothesisRecord, ...]', pair_affinities: 'tuple[HypothesisAffinity, ...]', diagnostics: 'Mapping[str, int | float]')¶

class panorai.object_localization.HypothesisRegionEvaluation(hypothesis_id: 'str', view_id: 'str', metrics: 'SphericalRegionMetrics')¶

class panorai.object_localization.HypothesisTransferEvaluation(hypothesis_id_a, hypothesis_id_b, view_id_a, view_id_b, role, forward_iou, reverse_iou, symmetric_iou)¶: Compact transfer score; transient ERP masks are intentionally omitted.

class panorai.object_localization.HypothesisViewConsistency(view_id_a, view_id_b, shared_track_count, union_track_count, track_jaccard, smaller_view_track_coverage, matched_feature_pairs, median_angular_reprojection_residual_deg=None, reprojection_agreement=None, embedding_cosine_similarity=None, embedding_source_ids=None, embedding_preprocessing_id=None)¶: Independent correspondence and optional appearance evidence for a pair.

class panorai.object_localization.HypothesisViewEmbedding(view_id, vector, source_id, preprocessing_id=None)¶: Optional externally computed descriptor for one hypothesis face.

class panorai.object_localization.HypothesisViewPairEvaluation(hypothesis_id: 'str', consistency: 'HypothesisViewConsistency')¶

class panorai.object_localization.JointMatchCluster(cluster_id, parent_association_id, match_indices, region_a, region_b, diameter_a_deg, diameter_b_deg)¶: One paired candidate instance supported by matched features.

class panorai.object_localization.JointMatchClusterConfig(max_neighbor_angle_deg=5.0, min_matches=3, max_clusters=None, pose_inliers_only=True)¶

Conservative gates for grouping matches in both spherical views.

Two matches are neighbours only when their bearing separation is within max_neighbor_angle_deg in both views. Connected components smaller than min_matches are reported as discarded support.

class panorai.object_localization.JointMatchClusterResult(parent_association_id, clusters, source_match_count, raw_cluster_count, retained_cluster_count, discarded_small_cluster_count, discarded_match_count, truncated_cluster_count, max_neighbor_angle_deg, pose_inliers_only, interface='panorai-joint-match-clusters/v1')¶: Retained paired regions and explicit clustering diagnostics.

class panorai.object_localization.MultiviewObjectHypothesis(hypothesis_id: 'str', concept_id: 'str', concept_name: 'str', vocabulary: 'str', state: 'MultiviewHypothesisState', view_ids: 'tuple[str, ...]', track_ids: 'tuple[str, ...]', proxy_ids: 'tuple[str, ...]', semantic_score: 'float', spatial: 'MultiviewSpatialRegion', observations: 'tuple[SemanticTrackObservation, ...]' = (), view_regions: 'tuple[HypothesisGnomonicRegion, ...]' = (), source_ids: 'tuple[str, ...]' = (), reasons: 'tuple[str, ...]' = (), provenance: 'Mapping[str, object]' = <factory>)¶

class panorai.object_localization.MultiviewObjectHypothesisResult(hypotheses: 'tuple[MultiviewObjectHypothesis, ...]', track_observations: 'tuple[SemanticTrackObservation, ...]', diagnostics: 'Mapping[str, int | float | str]', bearing_track_triangulations: 'tuple[BearingTrackTriangulation, ...]' = (), interface: 'str' = 'panorai-multiview-object-hypotheses/v1', stability: 'str' = 'experimental')¶

class panorai.object_localization.MultiviewSpatialRegion(mode, frame, units, support_kind, center_xyz, extent_covariance_xyz, bounding_box_xyz, extent_radius, support_points_xyz, support_weights, bearing_xyz=None, bearing_view_id=None)¶

Spatial support for one semantic hypothesis.

extent_covariance_xyz describes the spread of supporting reconstructed tracks. It is deliberately not called estimator uncertainty.

class panorai.object_localization.ObjectHypothesis(hypothesis_id: 'str', class_id: 'int', class_name: 'str', identity_state: 'IdentityState', view_ids: 'tuple[str, str]', region_ids: 'tuple[str, str]', association_id: 'str', semantic_score: 'float', identity_score: 'float', location: 'SpatialLocationHypothesis')¶

class panorai.object_localization.ObjectLocalizationConfig(min_region_matches=3, min_region_pose_inliers=3, max_median_pose_residual_deg=1.5, ambiguity_margin=0.05, require_accepted_pose=True, min_triangulated_points=2, min_triangulation_angle_deg=0.5, max_reprojection_error_deg=2.0)¶: Conservative gates for the simple two-view v1 workflow.

class panorai.object_localization.ObjectLocalizationPipeline(config=None)¶: Associate and localize semantic regions in exactly two views.

class panorai.object_localization.ObjectLocalizationResult(query, hypotheses, associations, failure_reasons=(), interface='panorai-object-localization/v1', stability='experimental', diagnostics=<factory>)¶: Association and spatial hypotheses returned for one textual query.

class panorai.object_localization.PairObjectLocalizationInput(view_id_a, view_id_b, query, regions_a, regions_b, features_a, features_b, matches, pose, translation_scale=None, metric_units='m')¶

All evidence needed by the two-view workflow.

The pose convention is point_b = R @ point_a + t. The optional translation_scale multiplies the pose’s unit translation direction.

class panorai.object_localization.ReducedHypothesisRecord(representative, source_hypothesis_ids, original_scores, reduced_score, action, fused_spatial=None, suppressed_by=())¶: A selected, fused, or softly decayed hypothesis record.

class panorai.object_localization.RegionAssociation(association_id: 'str', region_id_a: 'str', region_id_b: 'str', class_id: 'int', class_name: 'str', match_indices: 'np.ndarray', inlier_match_indices: 'np.ndarray', match_count: 'int', inlier_count: 'int', semantic_score: 'float', median_pose_residual_deg: 'float | None', ranking_score: 'float', state: 'AssociationState', reasons: 'tuple[str, ...]' = ())¶

class panorai.object_localization.RegisteredBearingObservation(view_id, feature_index, bearing_xyz, weight=1.0)¶

One feature bearing in a panorama with a known registered pose.

The bearing uses the panorama frame. It carries no range and therefore cannot imply a 3-D point on its own.

class panorai.object_localization.RegisteredBearingTrack(track_id, observations)¶: Cross-view correspondence track without an input 3-D point.

exception panorai.object_localization.RegisteredPoseInconsistencyError(*, input_track_count, nondegenerate_track_count, rejected_track_count, rejected_fraction, reason_counts)¶

Registered poses fail the systematic bearing-track consistency gate.

Local low-parallax or ill-conditioned tracks are reported as degeneracies and do not cause this exception. It is raised only when enough non-degenerate tracks exist and the configured fraction fails cheirality or angular reprojection. Structured counts let callers abstain and report pose_inconsistent without parsing an error string.

class panorai.object_localization.RegisteredRadialDepth(view_id, values, support, units, source_id, provenance=<factory>)¶

Radial range attached to one registered panorama.

Support is authoritative. Supported samples must be finite and strictly positive; unsupported numeric values are ignored and never become zero range.

class panorai.object_localization.RegisteredSemanticEvidence(view_id, channel_ids, scores, support, vocabulary, source_id, provenance=<factory>)¶

Dense semantic channels for one registered canonical ERP.

The class is intentionally NumPy-only. from_dense accepts the public DenseSemanticEvidence contract by attributes without importing the optional deep-learning or segmentation namespaces.

classmethod from_dense(view_id, evidence, *, source_id=None)¶: Adapt a DenseSemanticEvidence-compatible object lazily.

class panorai.object_localization.SemanticConcept(concept_id, display_name, proxy_weights, vocabulary='imagenet-1k')¶

Canonical concept supported by one or more named semantic channels.

proxy_weights names channels in RegisteredSemanticEvidence. Weights are normalized on construction, while absent channels contribute zero evidence rather than silently redistributing their mass.

class panorai.object_localization.SemanticMatchPrior(query_text, query_class_ids, used_class_ids, sampling_weights, semantic_support, valid_mask, supported_match_indices, effective_sample_size, max_to_min_valid_weight_ratio, fallback_reason, interface='panorai-semantic-match-prior/v1')¶

Full-length proposal weights and auditable semantic support.

Invalid match rows retain zero weight. Every valid row has positive weight, including rows outside all semantic regions, so the global geometric fallback remains reachable.

to_bearing_correspondences(matches)¶: Attach this proposal prior without changing match validity.

property weights¶: Alias matching the bearing-correspondence weight vocabulary.

class panorai.object_localization.SemanticMatchPriorConfig(uniform_mix=0.25, semantic_power=1.0, max_weight_ratio=4.0)¶

Conservative mixture used to turn shared CAM support into weights.

uniform_mix is a positive floor on every valid match. A value of one disables semantic preference exactly. max_weight_ratio independently caps concentration after the semantic mixture is formed.

class panorai.object_localization.SemanticQuery(text, class_ids, vocabulary='imagenet-1k')¶

Text query resolved to explicit classes in a declared vocabulary.

V1 does not guess how free text maps to classes. A caller may use a classifier, synonym table, or UI selector, but must preserve both the original text and the resolved vocabulary IDs.

class panorai.object_localization.SemanticRegionObservation(region_id, view_id, class_id, class_name, semantic_score, feature_indices, membership_weights, centroid_bearing=None, source_id=None)¶

One class-labelled region observed in one panorama.

feature_indices index the corresponding panorai.features.SphericalFeatureSet. The region keeps no descriptor copy, so matching provenance remains owned by the feature API.

class panorai.object_localization.SemanticRegionProposalConfig(threshold=0.5, connectivity=8, min_component_cells=1, min_feature_count=1, max_regions=None)¶: Topology and support gates for splitting one CAM into candidates.

class panorai.object_localization.SemanticRegionProposalResult(regions, map_shape_hw, threshold, connectivity, raw_component_count, retained_component_count, rejected_small_component_count, rejected_no_feature_count, truncated_component_count, seam_crossing_component_count, interface='panorai-semantic-region-proposals/v1')¶: Retained feature-indexed regions plus component rejection counts.

class panorai.object_localization.SemanticTrackObservation(view_id, track_id, concept_id, feature_index, supported, concept_score, proxy_scores, bearing_xyz, source_id, geometric_residual_deg=None)¶: One concept score sampled at one registered track observation.

class panorai.object_localization.SpatialLocationHypothesis(state: 'LocationState', mode: 'LocationMode', frame: 'str', units: 'str', position_xyz: 'np.ndarray | None', covariance_xyz: 'np.ndarray | None', bearing_xyz: 'np.ndarray | None', uncertainty_radius: 'float | None', supporting_match_indices: 'np.ndarray', triangulated_count: 'int', median_parallax_deg: 'float | None', median_reprojection_error_deg: 'float | None', location_score: 'float')¶

class panorai.object_localization.SphericalAnchorAffinity(anchor_id_a: 'str', anchor_id_b: 'str', hard_compatible: 'bool', hard_reasons: 'tuple[str, ...]', factors: 'Mapping[str, float]', available_association_cues: 'tuple[str, ...]', affinity: 'float')¶

class panorai.object_localization.SphericalAnchorCluster(cluster_id: 'str', concept_id: 'str', vocabulary: 'str', anchor_ids: 'tuple[str, ...]', view_ids: 'tuple[str, ...]', source_hypothesis_ids: 'tuple[str, ...]', proposal_family_ids: 'tuple[str, ...]', score: 'float', state: 'str', consensus_regions: 'tuple[HypothesisGnomonicRegion, ...]', minimum_pair_affinity: 'float | None')¶

class panorai.object_localization.SphericalAnchorClusterDecision(cluster, action, representative_cluster_id, minimum_view_iou)¶: One deterministic cluster-level NMS decision without retaining masks.

class panorai.object_localization.SphericalAnchorDecision(anchor, action, representative_anchor_id, solid_angle_iou)¶: One deterministic intra-view NMS decision without retaining a mask.

class panorai.object_localization.SphericalAnchorFusionConfig(intra_view_nms_iou=0.85, maximum_anchors_per_view_concept=6, pair_affinity_threshold=0.2, complete_link_threshold=0.2, cluster_nms_iou=0.55, minimum_views_for_confirmation=2, minimum_depth_coverage=0.25, center_reprojection_sigma_deg=1.0, maximum_center_reprojection_error_deg=2.0, consensus_threshold=0.5, consensus_coverage=0.9, consensus_margin_deg=3.0, minimum_final_fov_deg=8.0, maximum_final_fov_deg=120.0, evaluation_shape_hw=(128, 256), transfer_weight=1.0, track_weight=1.0, appearance_weight=1.0)¶: Frozen policy for anchor suppression, association, and FOV refitting.

class panorai.object_localization.SphericalAnchorFusionResult(retained_anchors: 'tuple[SphericalFOVAnchor, ...]', suppressed_anchor_ids: 'tuple[str, ...]', anchor_decisions: 'tuple[SphericalAnchorDecision, ...]', pair_affinities: 'tuple[SphericalAnchorAffinity, ...]', clusters: 'tuple[SphericalAnchorCluster, ...]', suppressed_cluster_ids: 'tuple[str, ...]', cluster_decisions: 'tuple[SphericalAnchorClusterDecision, ...]', diagnostics: 'Mapping[str, int | float | str]', interface: 'str' = 'panorai-spherical-anchor-fusion/v1', stability: 'str' = 'experimental')¶

class panorai.object_localization.SphericalAnchorPairCues(anchor_id_a, anchor_id_b, geometric_compatibility=None, transfer_iou=None, track_consistency=None, appearance_cosine_similarity=None, geometric_contradiction=False, depth_separated=False, track_contradiction=False, provenance=<factory>)¶

Optional, validity-explicit cues for one ordered anchor pair.

None means that a cue was unavailable. A numeric zero means that the cue was evaluated and found no agreement. Those states are intentionally distinct.

class panorai.object_localization.SphericalFOVAnchor(anchor_id, view_id, concept_id, vocabulary, region, score, track_ids=(), source_hypothesis_ids=(), provenance=<factory>, proposal_family_ids=())¶: One local object-region proposal expressed as a gnomonic FOV.

class panorai.object_localization.SphericalObjectHypothesisConfig(minimum_observation_score: 'float' = 0.25, minimum_track_score: 'float' = 0.25, minimum_independent_views: 'int' = 2, minimum_tracks_per_hypothesis: 'int' = 2, ambiguity_margin: 'float' = 0.05, maximum_ambiguous_track_fraction: 'float' = 0.5, maximum_track_residual_deg: 'float' = 2.0, minimum_triangulation_angle_deg: 'float' = 0.5, maximum_triangulation_reprojection_error_deg: 'float' = 2.0, maximum_triangulation_condition_number: 'float' = 100000000.0, minimum_positive_depth_fraction: 'float' = 1.0, minimum_tracks_for_pose_consistency_gate: 'int' = 4, maximum_pose_inconsistent_track_fraction: 'float' = 0.25, spatial_cluster_radius: 'float | None' = None, auto_cluster_radius_baseline_fraction: 'float' = 0.25, emit_bearing_only_fallbacks: 'bool' = True, metric_units: 'str' = 'm', gnomonic_output_shape_hw: 'tuple[int, int]' = (1024, 1024), minimum_gnomonic_fov_deg: 'float' = 70.0, maximum_gnomonic_fov_deg: 'float' = 150.0, gnomonic_margin_deg: 'float' = 8.0)¶

class panorai.object_localization.SphericalObjectHypothesisGenerator(config=None)¶

Generate conservative semantic hypotheses from registered tracks.

generate_from_bearings(*, poses, tracks, semantic_evidence, concepts, scale='arbitrary', reference_view_id=None)¶

Generate hypotheses when registered poses and tracks have no points.

Track points are estimated internally and retained only after geometric quality gates pass. A systematic set of non-degenerate failures is a pose-consistency error, not a collection of harmless angular fallbacks.

class panorai.object_localization.SphericalObjectHypothesisMethod(config=None)¶

Compose the paper method without owning semantic or depth inference.

run(*, semantic_evidence, concepts, reconstruction=None, poses=None, bearing_tracks=None, reconstruction_scale='arbitrary', reference_view_id=None, radial_depth=(), independent_regions_by_hypothesis=None, independent_region_provider=None, embeddings_by_hypothesis=None, pair_cues=(), anchor_pair_cues=(), geometric_quality_by_hypothesis=None, independent_view_factor_by_hypothesis=None)¶: Run every available stage with exactly one declared geometry source.

class panorai.object_localization.SphericalObjectHypothesisMethodConfig(generator: 'SphericalObjectHypothesisConfig' = <factory>, affinity: 'HypothesisAffinityConfig' = <factory>, anchor_fusion: 'SphericalAnchorFusionConfig' = <factory>, anchor_scale_factors: 'tuple[float, ...]' = (0.75, 1.0, 1.33), anchor_aspect_ratios: 'tuple[float, ...]' = (0.5, 1.0, 2.0), reprojection_sigma_deg: 'float' = 1.0, enable_registered_transfer: 'bool' = True, enable_anchor_fusion: 'bool' = True, enable_reduction: 'bool' = True)¶

class panorai.object_localization.SphericalObjectHypothesisMethodResult(generation: 'MultiviewObjectHypothesisResult', region_evaluations: 'tuple[HypothesisRegionEvaluation, ...]', view_pair_evaluations: 'tuple[HypothesisViewPairEvaluation, ...]', transfer_evaluations: 'tuple[HypothesisTransferEvaluation, ...]', anchor_fusion: 'SphericalAnchorFusionResult | None', reduction_pair_cues: 'tuple[HypothesisPairCues, ...]', reduction: 'HypothesisReductionResult | None', diagnostics: 'Mapping[str, int | float | str]')¶

class panorai.object_localization.SphericalRegionMetrics(view_id, concept_id, semantic_mass_recall, observed_sphere_fraction, mean_score_enrichment, supported_evidence_mass, region_evidence_mass, supported_solid_angle_weight, region_solid_angle_weight, region_pixel_count, supported_region_pixel_count)¶: Solid-angle-aware utility measurements for one evidence-bearing region.

class panorai.object_localization.SymmetricRegionTransfer(forward, reverse, symmetric_solid_angle_iou)¶: Two independently directed transfers and their symmetric score.

panorai.object_localization.assign_region_associations(proposals, config=None)¶: Apply a global one-to-one assignment and expose ambiguous choices.

panorai.object_localization.build_semantic_match_prior(query, regions_a, regions_b, matches, config=None)¶

Build query-conditioned weights from feature-indexed semantic regions.

Multiple regions or query classes cannot inflate a match by duplication: evidence is combined with a maximum both within a class and across query classes. The geometric mean requires support in both views.

panorai.object_localization.cluster_region_association_matches(association, evidence, config=None)¶

Split one accepted broad region pair using match proximity in both views.

This function does not assign a persistent identity. It returns paired SemanticRegionObservation candidates that can be passed back to ObjectLocalizationPipeline. Angular distances are computed from 3D bearings, so neighbourhoods cross the ERP longitude seam correctly.

panorai.object_localization.evaluate_spherical_region(evidence, concept, region)¶: Evaluate semantic mass, angular cost, and enrichment over support.

panorai.object_localization.fuse_registered_spherical_anchors(anchors, *, poses, radial_depth=(), pair_cues=(), config=None)¶: Associate per-view anchors and refit registered multiview consensus FOVs.

panorai.object_localization.generate_spherical_fov_anchors(base_region, *, concept_id, vocabulary, score, anchor_prefix, scale_factors=(0.75, 1.0, 1.33), aspect_ratios=(0.5, 1.0, 2.0), minimum_fov_deg=8.0, maximum_fov_deg=120.0, source_hypothesis_ids=(), proposal_family_id=None)¶: Generate deterministic scale/aspect alternatives around one direction.

panorai.object_localization.hypothesis_pair_affinity(first, second, *, cues=None, config=None)¶: Compute hard gates and the available-factor product from Eq. 41.

panorai.object_localization.hypothesis_view_consistency(hypothesis, embeddings=(), *, reprojection_sigma_deg=1.0)¶

Compare every view pair using registered tracks and optional embeddings.

Embeddings are intentionally caller-owned: PanorAi neither selects nor loads an encoder here. This keeps the core result useful with local descriptors, learned face embeddings, or no appearance backend at all.

panorai.object_localization.localize_region_association(association, evidence, config=None)¶: Triangulate regional inliers or retain an honest angular hypothesis.

panorai.object_localization.object_hypothesis_id(view_id_a, region_id_a, view_id_b, region_id_b)¶: Return the direction-independent ID for one two-view object pair.

panorai.object_localization.propose_region_associations(evidence, config=None)¶: Score every class-compatible region pair without assigning identities.

panorai.object_localization.reduce_object_hypotheses(candidates, *, pair_cues=(), config=None)¶: Fuse affinity-graph components and retain Soft-NMS alternatives.

panorai.object_localization.semantic_region_from_map(score_map, *, erp_shape_hw, features, region_id, class_id, class_name, threshold=0.5, semantic_score=None, source_id=None)¶

Index panorama features inside one CAM/semantic region score map.

The caller supplies one map per candidate region. V1 deliberately does not decide how a class map is split into instances. Pixel-center nearest sampling maps ERP feature coordinates to maps of any positive resolution.

panorai.object_localization.semantic_regions_from_map(score_map, *, erp_shape_hw, features, region_id_prefix, class_id, class_name, config=None, semantic_score=None, source_id=None)¶

Split one CAM into deterministic ERP-seam-aware region candidates.

Horizontal neighbours wrap periodically; vertical neighbours do not. Components are ordered by peak, activation mass, cell count, then raster anchor. Only components with enough CAM cells and indexed features are returned.

panorai.object_localization.spherical_anchor_pair_affinity(first, second, *, cues=None, config=None)¶: Combine alternative association cues without confusing missing and zero.

panorai.object_localization.spherical_region_mask(spec, shape_hw, *, chunk_rows=256)¶: Rasterize a gnomonic footprint on a canonical ERP by pixel membership.

panorai.object_localization.spherical_solid_angle_weights(shape_hw)¶: Return exact ERP pixel-cell area weights from the spherical core.

panorai.object_localization.symmetric_spherical_region_transfer(first_region, second_region, first_depth, second_depth, first_pose, second_pose)¶: Evaluate symmetric overlap using two independently inferred regions.

panorai.object_localization.transfer_spherical_region(source_region, target_region, source_depth, source_pose, target_pose, *, target_support=None)¶: Transfer one region through radial depth into another registered view.

panorai.object_localization.triangulate_registered_bearing_track(track, poses, config=None)¶

Estimate one latent world point from registered panorama bearings.

Poses follow x_v = R_v @ (X - C_v). A panorama bearing is transformed to the world ray d_v = R_v.T @ b_v. The estimate minimizes weighted squared distance to those rays. It is returned only when parallax, conditioning, cheirality, and angular reprojection gates all pass.