Spherical visual SLAM¶

panorai.slam is an Experimental, visual-only, arbitrary-scale SLAM surface. It provides two deliberately separate workflows:

SphericalIncrementalSLAM consumes central equirectangular images and returns one tracking result immediately per frame. It owns automatic keyframes, a local landmark map, spherical 2D–3D pose refinement, local bundle adjustment, relocalization, loop discovery, and global correction.
SphericalVisualSLAM consumes one calibrated central equidistant fisheye lens. It constructs the temporal graph incrementally but performs global reconstruction only when finish() freezes that graph.

Both workflows use PanorAi spherical features, relative pose, and global reconstruction. The implementation is NumPy/SciPy/OpenCV based; rosbags is needed only by dataset adapters.

Incremental central-ERP workflow¶

from panorai.slam import SphericalIncrementalSLAM

slam = SphericalIncrementalSLAM.from_preset("sift-flann")
for frame_id, timestamp_s, panorama_rgb in frames:
    tracking = slam.add_frame(
        panorama_rgb,
        timestamp_s=timestamp_s,
        frame_id=frame_id,
    )
    if tracking.pose is None:
        print(frame_id, tracking.reasons)
    else:
        use_online_pose(tracking.pose)

result = slam.finish()

The input must be a central ERP panorama. Each accepted relative pose is quality-gated. Landmark associations use only descriptor matches that are also geometric inliers. tracking.state is one of initializing, tracking, keyframe, relocalized, or lost. A lost frame has no pose; PanorAi does not fill trajectory gaps with fabricated estimates.

The local map triangulates unit bearings and maintains at most one active observation per frame and landmark. Local BA minimizes spherical log-map residuals, fixes the two oldest local poses to retain the monocular gauge, and removes observations above the configured angular limit. Accepted old/new keyframe links trigger loop correction through the global spherical mapper.

Calibrated-fisheye graph workflow¶

The first supported physical camera is one calibrated central equidistant fisheye lens:

from panorai.slam import EquidistantFisheyeCamera, SphericalVisualSLAM

camera = EquidistantFisheyeCamera.from_kalibr_yaml(
    "kalibr_imucam_chain.yaml",
    camera_id="cam0",
)
slam = SphericalVisualSLAM(camera)

for frame_id, timestamp_s, image in frames:
    slam.add_frame(image, timestamp_s=timestamp_s, frame_id=frame_id)

result = slam.finish()
if result.success:
    centers = result.centers_world
else:
    print(result.failure_reasons)

In this workflow every supplied frame is treated as a keyframe. temporal_window controls which recent keyframes are matched. The front end is incremental, while rotation averaging, track construction, positioning, triangulation, and spherical bundle adjustment run when finish() freezes the graph.

Coordinate and scale contract¶

Input pixels use (x, y) with a top-left origin. Kalibr/OpenCV equidistant intrinsics are inverted before estimation, and returned unit bearings use the PanorAi +X right, +Y up, +Z forward convention. Camera poses obey

\[x_i = R_i(X-C_i).\]

Monocular translation scale is unobservable. result.scale is therefore "arbitrary"; metric trajectories must be aligned with a similarity transform for evaluation.

Dual-fisheye limit¶

A dual-fisheye camera normally has two distinct optical centres. This version does not concatenate both lenses and pretend that they form one central camera. Use one calibrated lens, or supply a pre-stitched ERP only when its central-camera approximation is justified. A future generalized-camera back end is required for exact joint use of both lenses.

Current limits shared by both workflows include rolling-shutter correction, IMU fusion, metric scale, dynamic-scene modeling, and bounded-memory marginalization. The incremental ERP workflow has automatic keyframes, relocalization and loop discovery, but its current local BA window drops old optimization variables rather than marginalizing them into a prior. Failures are explicit. A final incremental result may contain a valid partial map while complete_trajectory remains false in evaluation tooling.

The current promotion decision is explicitly negative. API-006 has one strict-accurate three-panorama replay. PERF-003 measured eight valid three-frame runs at roughly 72.8–79.2 seconds per registered frame and 739–762 MiB peak RSS on its reference machine. That is mechanism evidence, not continuous-video, real-time, or bounded-memory evidence. Before another stability decision, evaluate long sequences first, then loop/relocalization, cross-camera and cross-dataset ATE/RPE, and finally sustained latency plus map memory growth.

API¶

Experimental calibrated-fisheye visual SLAM.

The first version is a visual-only, arbitrary-scale pipeline for one central camera. It deliberately does not merge non-coincident dual-fisheye lenses into a fictitious central panorama.

class panorai.slam.EquidistantFisheyeCamera(image_shape_hw, focal_xy, principal_xy, distortion, camera_id='cam0', max_theta_deg=100.0)¶

OpenCV/Kalibr pinhole-equidistant camera with PanorAi output axes.

Raster coordinates use a top-left origin and (x, y) order. OpenCV’s local camera coordinates are converted to PanorAi’s +X right, +Y up, +Z forward convention before rays leave this object.

image_shape_hw¶

focal_xy¶

principal_xy¶

distortion¶

camera_id¶

max_theta_deg¶

classmethod from_kalibr_yaml(path, *, camera_id='cam0', max_theta_deg=100.0)¶: Load one pinhole/equidistant camera from Kalibr YAML.

pixels_to_rays(pixels_xy)¶: Invert the equidistant model for arbitrary pixel coordinates.

to_dict()¶

valid_pixel_mask(*, edge_margin_px=0)¶: Return geometric support, optionally eroded by a raster margin.

class panorai.slam.FisheyeRayProjection(rays_xyz, valid)¶

Camera rays and validity produced from fisheye raster coordinates.

rays_xyz¶

valid¶

class panorai.slam.SphericalImageFrame(frame_id, timestamp_s, image)¶

One timestamped calibrated-fisheye image supplied to SLAM.

frame_id¶

timestamp_s¶

image¶

class panorai.slam.SphericalIncrementalSLAM(*, feature_pipeline=None, relative_pose_estimator=None, options=None)¶

Experimental online visual SLAM for central ERP images.

add_frame returns a pose/status immediately. Descriptor tracking, landmark association, keyframe selection and local BA are incremental. Global mapping is used only for accepted loop correction and final graph refinement; it is not the mechanism that creates online poses.

add_features(features, *, timestamp_s)¶: Advanced incremental route for already extracted spherical features.

add_frame(panorama, *, timestamp_s, frame_id=None, validity_mask=None)¶: Extract spherical features and increment the map by one ERP frame.

property current_pose¶

describe()¶

finish()¶: Freeze the session and optionally refine the complete keyframe graph.

classmethod from_preset(preset='sift-flann', *, face_sampler='cube', face_fov_deg=100.0, face_shape_hw=(320, 320), max_features=2048, options=None, relative_pose_estimator=None)¶

property local_map¶

reset()¶: Clear all trajectory and map state while retaining configuration.

run(frames)¶

class panorai.slam.SphericalIncrementalSLAMDiagnostics(input_frame_count: 'int', tracked_frame_count: 'int', lost_frame_count: 'int', keyframe_count: 'int', map_point_count: 'int', active_map_point_count: 'int', relocalization_count: 'int', loop_closure_count: 'int', pairwise_edge_count: 'int', local_ba_reports: 'tuple[SphericalLocalBAReport, ...]', stage_messages: 'tuple[str, ...]' = ())¶

input_frame_count¶

tracked_frame_count¶

lost_frame_count¶

keyframe_count¶

map_point_count¶

active_map_point_count¶

relocalization_count¶

loop_closure_count¶

pairwise_edge_count¶

local_ba_reports¶

stage_messages¶

to_dict()¶

class panorai.slam.SphericalIncrementalSLAMOptions(local_window_size=6, min_frame_features=64, min_pair_matches=20, min_map_correspondences=8, max_tracking_pose_candidates=2, keyframe_min_interval_s=0.1, keyframe_max_interval_s=1.0, keyframe_min_parallax_deg=1.0, keyframe_min_tracked_ratio=0.65, initial_baseline=1.0, min_triangulation_angle_deg=0.5, max_reprojection_error_deg=2.0, local_ba_loss='cauchy', local_ba_loss_scale_deg=1.0, local_ba_max_nfev=50, local_ba_max_points=512, loop_min_keyframe_separation=8, loop_min_matches=30, max_loop_candidates=5, optimize_global_on_loop=True, global_refine_on_finish=True, edge_admission='accepted', mapper=<factory>)¶

Policy for central-ERP incremental tracking and local mapping.

local_window_size¶

min_frame_features¶

min_pair_matches¶

min_map_correspondences¶

max_tracking_pose_candidates¶

keyframe_min_interval_s¶

keyframe_max_interval_s¶

keyframe_min_parallax_deg¶

keyframe_min_tracked_ratio¶

initial_baseline¶

min_triangulation_angle_deg¶

max_reprojection_error_deg¶

local_ba_loss¶

local_ba_loss_scale_deg¶

local_ba_max_nfev¶

local_ba_max_points¶

loop_min_keyframe_separation¶

loop_min_matches¶

max_loop_candidates¶

optimize_global_on_loop¶

global_refine_on_finish¶

edge_admission¶

mapper¶

to_dict()¶

class panorai.slam.SphericalIncrementalSLAMResult(success, failure_reasons, frames, keyframes, map_points, diagnostics, options, global_reconstruction=None, interface='panorai-spherical-incremental-slam/v1', scale='arbitrary')¶

Final snapshot of an incremental arbitrary-scale spherical map.

success¶

failure_reasons¶

frames¶

keyframes¶

map_points¶

diagnostics¶

options¶

global_reconstruction¶

interface¶

scale¶

describe()¶

pose(frame_id)¶

property trajectory¶

class panorai.slam.SphericalKeyframeSummary(frame_id: 'str', timestamp_s: 'float', pose: 'SphericalSLAMPose', feature_count: 'int', active_landmark_count: 'int')¶

frame_id¶

timestamp_s¶

pose¶

feature_count¶

active_landmark_count¶

class panorai.slam.SphericalLocalBAReport(keyframe_ids: 'tuple[str, ...]', point_count: 'int', observation_count: 'int', variable_pose_count: 'int', initial_cost: 'float', final_cost: 'float', removed_observation_count: 'int', success: 'bool')¶

keyframe_ids¶

point_count¶

observation_count¶

variable_pose_count¶

initial_cost¶

final_cost¶

removed_observation_count¶

success¶

class panorai.slam.SphericalMapObservation(frame_id, feature_index, bearing_xyz, active=True, residual_deg=0.0)¶

One keyframe bearing associated with a local-map point.

frame_id¶

feature_index¶

bearing_xyz¶

active¶

residual_deg¶

class panorai.slam.SphericalMapPoint(point_id, position_world, observations, active=True)¶

Immutable public snapshot of one arbitrary-scale local landmark.

point_id¶

position_world¶

observations¶

active¶

class panorai.slam.SphericalSLAMDiagnostics(input_frame_count: 'int', keyframe_count: 'int', dropped_frames: 'tuple[tuple[str, str], ...]', candidate_pair_count: 'int', retained_pair_count: 'int', pair_match_counts: 'tuple[tuple[tuple[str, str], int], ...]', backend_name: 'str', backend_version: 'str')¶

input_frame_count¶

keyframe_count¶

dropped_frames¶

candidate_pair_count¶

retained_pair_count¶

pair_match_counts¶

backend_name¶

backend_version¶

to_dict()¶

class panorai.slam.SphericalSLAMFrameSummary(frame_id: 'str', timestamp_s: 'float', feature_count: 'int', checksum_sha256: 'str')¶

frame_id¶

timestamp_s¶

feature_count¶

checksum_sha256¶

class panorai.slam.SphericalSLAMOptions(temporal_window=2, min_features_per_frame=64, min_matches_per_pair=16, edge_margin_px=12, extractor=<factory>, matcher=<factory>, mapper=<factory>)¶

Front-end and global-back-end policy for a visual SLAM session.

temporal_window¶

min_features_per_frame¶

min_matches_per_pair¶

edge_margin_px¶

extractor¶

matcher¶

mapper¶

to_dict()¶

class panorai.slam.SphericalSLAMPose(frame_id: 'str', timestamp_s: 'float', rotation_world_to_camera: 'np.ndarray', center_world: 'np.ndarray')¶

frame_id¶

timestamp_s¶

rotation_world_to_camera¶

center_world¶

property R¶

property center¶

class panorai.slam.SphericalSLAMResult(success, failure_reasons, poses, keyframes, reconstruction, diagnostics, options, interface='panorai-spherical-slam/v1', scale='arbitrary')¶

Immutable arbitrary-scale SLAM trajectory and sparse reconstruction.

success¶

failure_reasons¶

poses¶

keyframes¶

reconstruction¶

diagnostics¶

options¶

interface¶

scale¶

property centers_world¶

describe()¶

pose(frame_id)¶

class panorai.slam.SphericalTrackingResult(frame_id, timestamp_s, state, pose, reference_frame_id, is_keyframe, feature_count, match_count, inlier_count, map_correspondence_count, median_parallax_deg, local_map_point_count, reasons=())¶

Online result returned immediately for one ERP input frame.

frame_id¶

timestamp_s¶

state¶

pose¶

reference_frame_id¶

is_keyframe¶

feature_count¶

match_count¶

inlier_count¶

map_correspondence_count¶

median_parallax_deg¶

local_map_point_count¶

reasons¶

class panorai.slam.SphericalVisualSLAM(camera, *, options=None, backend=None, relative_pose_estimator=None)¶

Experimental calibrated-fisheye visual SLAM session.

add_frame incrementally extracts and matches features. finish runs PanorAi’s global spherical mapper over the retained temporal graph. Every input frame is a keyframe; callers control temporal sampling explicitly.

add_frame(image, *, timestamp_s, frame_id=None)¶: Extract one frame and connect it to the recent keyframe window.

finish()¶: Freeze the temporal graph and run global spherical reconstruction.

reset()¶: Clear session state while retaining immutable configuration.

run(frames)¶: Process a finite frame stream and finalize its global map.