Tutorial: resize-free spherical monocular depth¶

Field	Contract
Application	Estimate radial range from one panorama
Input	One native 2:1 ERP
Output	Radial range, validity, and model/port provenance
Strategy	Sphere-native CNN or projection-domain ViT control
API	`panorai.experimental.deep_learning` depth adapters
Stability	Experimental
Dependencies	`panorai[deep-learning-depth]`; external verified weights
Evidence	Model-port integration evidence with stated real-scene limits

How this application is composed¶

The CNN route composes spherical learned convolution/pooling layers and a typed continuous radial-range output. The ViT control composes gnomonic projection, the unchanged planar model, axial-to-radial conversion and validity-aware depth fusion. Use the model-specific loaders: they wrap external artifact checks, conversion coverage and provenance that a bare SphericalConv2d stack cannot establish on its own.

This tutorial exposes two complementary stages of PanorAi’s monocular-depth portability work as Experimental, opt-in APIs. The CNN path loads an exact external Metric3D-v1 ConvNeXt/Hourglass model, ports its learned spatial layers to PanorAi spherical convolution, and returns radial range on the input ERP lattice. The ViT control keeps the official DA3Metric-Large network unchanged, evaluates native-density gnomonic views without resizing, converts its axial output to radial range, and reconstructs the predictions in ERP.

The interfaces are panorai-spherical-metric-depth/v1-experimental and panorai-depth-anything-3-metric-tangent/v1-experimental. Neither is part of stable geometry. They do not train models or bundle upstream code or weights.

See the Image processing theme for the distinction between sphere-native learned sampling and a planar model evaluated through projected views.

PanorAi-specific: exact artifact identity, explicit acquisition consent, complete layer/parameter provenance, differentiable longitude-wrapped tangent sampling, bounded-memory row chunking, resize-free ERP input, and radial-range semantics in the canonical panorama frame.

1. Install the optional runtime¶

pip install "panorai[deep-learning-depth]"

The ordinary panorai, panorai.geometry, and panorai.experimental imports remain independent of Torch and Metric3D. Merely importing the deep-learning module never contacts the network. Acquisition occurs only in the explicit function below.

2. Review the external boundary¶

Two CNN capacities are supported by the same porting contract. The Large variant is used below; Tiny remains useful for lower-memory execution.

component	pinned identity	SHA-256	terms
Metric3D source	commit `fd90d0dad05d913c4f45ed7a9789b8f1dfde22e0`	`929432fd1f7d1f2c45f51407e4ef30ca5758eb207b15e51c9b480048176ee365`	BSD-2-Clause source
ConvNeXt-Tiny/Hourglass v1 checkpoint	Hugging Face revision `80d2d1410afb4b23cd9d18c6be9144483d4b70b6`	`bc41f5f919bb0388bbc88fe1d9e60b49b826c620b4acc1b6c10f473f6d4741a5`	no separate checkpoint/model-card license found
ConvNeXt-Large/Hourglass v1 checkpoint	Hugging Face revision `80d2d1410afb4b23cd9d18c6be9144483d4b70b6`	`0eaaa2501557ac627ada0070e257c6bc74e3e60b45477b29c2efceb70440cfe8`	no separate checkpoint/model-card license found

The missing checkpoint terms are a redistribution blocker. Setting accept_upstream_terms=True records that the caller reviewed this boundary; it is not a license grant from PanorAi. Source, checkpoint, extraction, manifest, and predictions stay in the external cache selected by the caller.

3. Acquire and port the model¶

from pathlib import Path

from panorai.experimental.deep_learning import (
    load_spherical_metric3d_convnext_large_v1,
)

loaded = load_spherical_metric3d_convnext_large_v1(
    accept_upstream_terms=True,
    cache_dir=Path("/data/model-cache/panorai-metric3d"),
    device="cpu",
    max_sampled_elements=100_000_000,
)

print(loaded.assets.to_dict())
print(loaded.port_report.to_dict())

Acquisition verifies full file size and SHA-256 before extraction or deserialization. The checkpoint is read with PyTorch’s weights_only=True. A corrupt cached file is rejected and is not silently deleted or replaced.

For the Large model, the port report identifies 66 replaced spatial operations: 63 direct spherical convolutions and three spherical resize-convolution upsamplers. It contains 203,243,090 parameters and no remaining planar spatial layer. Direct convolution ports retain the same Parameter objects. The three stride-two transposed-convolution kernels are flipped, channel-transposed and divided by the stride area before spherical convolution; this deterministic resize_conv policy suppresses the period-eight bands produced by a stacked zero-insertion decoder. It is not numerically equivalent to the source transposed convolution, and the port report records that distinction. A non-empty remaining_planar_spatial_layers is a hard failure.

max_sampled_elements bounds the temporary tangent-neighbour tensor by processing output rows in chunks. It does not resize the ERP, change kernel weights of direct ports, or alter the global output lattice.

4. Infer without resizing¶

import numpy as np
from PIL import Image
import torch

rgb = np.asarray(Image.open("panorama.png").convert("RGB"), dtype=np.float32)
height, width = rgb.shape[:2]
assert width == 2 * height
assert height % 32 == 0 and width % 32 == 0

values = torch.from_numpy(rgb).permute(2, 0, 1).unsqueeze(0)
with torch.inference_mode():
    radial_range_m = loaded.model(values)

assert tuple(radial_range_m.shape) == (1, 1, height, width)

The public high-resolution run preserves the complete 1024×2048 lattice:

Metric3D Large sphere-native radial range at 1024×2048

Reproduce it from the verified CC0 2K HDRI and external model cache:

python scripts/generate_metric3d_high_resolution_preview.py \
  --source-hdr /data/poly_haven_studio_2k.hdr \
  --metric3d-cache /data/model-cache/panorai-metric3d

The tracked JPEG is a PanorAi-generated visualization over the CC0 input. It contains no checkpoint bytes. PanorAi does not claim that the BSD-2-Clause code license covers the separately hosted weights and does not redistribute them.

Input is torch.float32 NCHW RGB in the explicit range [0, 255] on the same device as the model. The wrapper does not resize, crop, pad, or prefilter. This is intentional: resizing would add a minification/aliasing variable that the current experiment has not controlled well enough. If the native ERP dimensions are not divisible by 32, the caller must choose and document a dataset-side sampling policy; the loader refuses to invent one.

The model’s canonical output is converted with effective ERP focal length width / (2π) and clamped to its declared 0.3–150 m domain. The result is radial range, so a 3D point is X = range * ray in PanorAi’s canonical +X right, +Y up, +Z forward frame. It is not camera z-depth.

Model prediction coverage is not data validity. A scanner blind region, missing RGB, or missing reference range still requires a separate explicit boolean mask; neither black pixels nor the 0.3 m prediction floor imply invalidity.

5. What spherical porting means¶

At each output ray, PanorAi lays the pretrained raster kernel in the local east/north tangent basis, maps taps to the unit sphere with the exponential map, and samples the ERP differentiably with longitude wrapping. SphericalConvTranspose2d is the exact bilinear-sampler adjoint. Metric3D’s three learned upsamplers instead use the explicit artifact-resistant resize_conv policy documented above.

This procedure preserves learned numbers; it does not prove that a perspective-trained feature is rotation invariant or semantically calibrated on panoramas. Resolution also changes angular tap spacing and effective receptive field. A native-resolution run is therefore a distinct experiment, not merely a higher-quality rendering.

6. Current evidence and limits¶

The included public prediction is qualitative execution evidence, not ground truth or an accuracy result. Public accuracy evidence remains an open requirement for this Experimental adapter.

The loader and port report therefore prove only mechanical availability, complete learned-layer replacement, and identity preservation of the original weights. They do not prove that those outdoor-trained weights retain useful semantics on an ERP, that the spherical and planar/cubemap computations are numerically equivalent, or that the model is suitable for a particular domain.

The first public adapter deliberately excludes:

hidden resize or antialiasing policy;
dataset loading, training, fine-tuning, or checkpoint redistribution;
a tangent-view oracle;
ViT attention or positional-embedding conversion;
calibrated uncertainty or automatic validity masks.

The DA3 control below makes the CNN and ViT portability comparison reproducible under the same radial-range, explicit-mask, native-resolution, view-projection, and 3D structure protocol. It does not turn the ViT into a sphere-native network.

7. Depth Anything 3 Metric Large tangent control¶

DA3Metric-Large is available as a separate reproducibility adapter. Its artifact boundary is fully explicit:

component	pinned identity	SHA-256	terms
Depth Anything 3 source	commit `3d835ec1a5802d64a8b8b15f817a1ab54809bfe4`	`98aa2dd53ab44b96cef5190ae4841c6ae51797d099b784e08e12a1a55bd3a69b`	Apache-2.0
DA3Metric-Large checkpoint	Hugging Face revision `4010e39f3634a45bc60553321fb49fb760bd594e`	`bbea5b0b3ee389849cffa7ddae89de064a90abd2b055fc5aa99aac68db324776`	Apache-2.0 model card

Source and checkpoint remain outside the source tree, wheel, and sdist. The adapter builds the architecture declared by the pinned da3metric-large.yaml, loads the safetensors checkpoint strictly, freezes the network, and bypasses the official high-level resize path.

from pathlib import Path

import numpy as np
from PIL import Image

from panorai.data import EquirectangularImage
from panorai.experimental.deep_learning import load_da3metric_large

rgb = np.asarray(Image.open("panorama.png").convert("RGB"), dtype=np.uint8)
panorama = EquirectangularImage(rgb)

# 42 overlapping views. H and W must each be divisible by DA3's patch size 14.
views = panorama.views(
    "icosahedron",
    subdivisions=1,
    size=(812, 1400),
    fov=(100.0, 80.0),
)

loaded_da3 = load_da3metric_large(
    accept_upstream_terms=True,
    cache_dir=Path("/data/model-cache/panorai-da3"),
    device="cuda",
)

# All views above share shape and FOV, hence the same intrinsic matrix.
adapter = loaded_da3.tangent(views[0].spec)
predicted = views.map(adapter, input="image", output="depth", units="m")
erp_depth = predicted.reconstruct(
    blend={"depth": "gaussian"},
    modalities=("depth",),
)

radial_range_m = erp_depth.depth
radial_valid = erp_depth.validity("depth")

The adapter accepts only HWC uint8 or float32 RGB in [0, 255] at the declared tangent shape. It does not resize, crop, pad, clamp, prefilter, or smooth. The exact conversion for pixel centre (u, v) is

z_m = network_depth * ((fx + fy) / 2) / 300
range_m = z_m * sqrt(1 + ((u + 0.5 - W/2) / fx)^2
                           + ((v + 0.5 - H/2) / fy)^2)

Here z_m is axial depth along the virtual-camera forward axis and range_m is the radial quantity required by PanorAi’s geometry contract. Finite positive model outputs define numerical prediction validity; the workflow intersects that mask with each view’s geometric support. It never infers validity from black RGB or from a zero-filled missing region.

The final Gaussian blend in this example is intentionally a fixed baseline. It does not claim that view boundaries are solved. A learned overlap fusion conditioned by corresponding view tokens remains a separate research module, not part of this adapter or the projection core.