Tutorial: pretrained spherical FCN and class-activation maps

Field	Contract
Application	Localize semantic regions
Input	One ERP and a frozen pretrained classifier/channel
Output	Dense directional CAM evidence and optional FOV proposal
Strategy	Sphere-native learned evidence
API	panorai.experimental.deep_learning
Stability	Experimental
Dependencies	panorai[deep-learning]; external verified weights
Evidence	Real CC0 output; CAM is not a mask or calibrated probability

How this application is composed

The port replaces learned planar spatial layers with panorai.core.torch spherical convolution/pooling while retaining ordinary pointwise layers and the pretrained parameter objects. Solid-angle reduction produces global scores; retaining the dense class lattice produces CAM evidence. The model loaders and proxy facade are wrappers around that composition because artifact identity, layer conversion and provenance must be checked together.

This tutorial ports three Torchvision ImageNet classifiers to differentiable spherical inference while preserving their learned parameters. The API is Experimental and opt-in under panorai.experimental.deep_learning; it is not part of the stable projection contract.

See the Image processing theme for the distinction between sphere-native sampling and planar models evaluated through projected views.

PanorAi-specific: full spatial-layer inventory, exact parameter reuse, longitude-wrapped tangent sampling, spherical-area score pooling, checkpoint provenance, and explicit separation between CAM localization and semantic segmentation.

Industrial application card: directional evidence, not segmentation

This real output uses the public CC0 Boiler Room ERP at 512×1024. The exact ImageNet class prison (index 743), declared beforehand as the grid-like-structure proxy, ranked first with classifier probability 0.1920. The left panel shows a within-channel blue–yellow–red CAM with its colorbar. The right panel is explicitly a method diagram: independent region/FOV proposals in registered views can support one shared spatial hypothesis. It is not a measured second-view match. The native evidence lattice is only 16×32; interpolation is display-only.

This is a deliberately readable channel, not cherry-picked proof of object recognition. The complete proxy-channel audit—including weak and failed channels—is documented in the industrial comparison and benchmark commands below. The component and FOV are region hypotheses, not a pipe/panel segmentation, ground truth, or calibrated pixel probability. Source, model, port, and derivative hashes are in image provenance and application-gallery-metadata.json.

Cross-model regression comparison on the forest fixture

The following outputs use the tracked CC0 Nature Reserve Forest ERP at its native 512×1024 resolution. All three backbones generated a 16×32 dense lattice. The overlays interpolate that lattice back to the ERP for inspection; interpolation does not create additional spatial evidence. Red means larger min-max-normalized evidence within that channel, not per-pixel probability.

ImageNet ResNet18: fixed class lakeside

This is a classical fixed ImageNet class channel. It demonstrates the object-oriented closed-vocabulary path and ranked third on this panorama; the map is weak evidence, not a claim that the panorama contains a labelled lake.

Places365 ResNet18: fixed scene forest/broadleaf

The Places365 scene channel concentrates on several vegetation masses. This is the closed-vocabulary scene extension, using the same frozen-weight spherical portability strategy.

OpenCLIP RN50: prompt a photo of a path

The prompt-defined channel shifts its stronger responses toward the ground. It is local image-text similarity from the dense reinterpretation, not the original global CLIP score and not a segmentation mask.

1. Install the optional stack

Install Torch and Torchvision only when this experiment is required:

pip install "panorai[deep-learning]"

The ordinary panorai, panorai.geometry, and panorai.experimental imports remain independent of Torchvision. Importing the explicit deep-learning submodule requires Torch, while Torchvision is loaded lazily when a pretrained model is requested.

Supported official DEFAULT weights are:

architecture	perspective crop	FCN head conversion
AlexNet	224×224	first linear layer → valid 6×6 convolution
VGG16	224×224	first linear layer → valid 7×7 convolution
ResNet18	224×224	final linear layer → 1×1 convolution

2. Download automatically or prefetch

No checkpoint is stored in the repository or PanorAi distribution. The first inference request downloads a missing checkpoint through Torchvision into the user-controlled Torch cache. PanorAi verifies the URL hash prefix and records the complete SHA-256, path, size, URL, and whether the file was already cached.

Prefetch all three models before an offline run:

python benchmarks/spherical_fcn_cam/download_models.py

Or prefetch a subset:

python benchmarks/spherical_fcn_cam/download_models.py \
  --model resnet18 --model vgg16

The command emits JSON with schema panorai-pretrained-imagenet-cache/v1. A cached file whose SHA-256 no longer matches the hash prefix in the official URL is rejected rather than silently used or deleted. Remove or replace such a file deliberately before retrying.

The same behavior is available from Python through prefetch_imagenet_weights() and load_pretrained_imagenet_model(). These helpers contact the network only when their explicit model-loading operation finds a missing checkpoint.

from panorai.experimental.deep_learning import prefetch_imagenet_weights

records = prefetch_imagenet_weights(("alexnet", "vgg16", "resnet18"))
for record in records:
    print(record.model_name, record.cache_path, record.sha256)

Checkpoint licenses and terms remain those of their upstream publishers. The prefetch command is acquisition by the user, not redistribution by PanorAi.

3. What is ported

For AlexNet and VGG16, ImageNetFCN reshapes each linear classifier parameter into the corresponding convolution parameter. For ResNet18, the final linear classifier becomes a 1×1 convolution and the global average is moved after dense prediction.

port_module_with_report() then recursively performs:

Conv2d    → SphericalConv2d
MaxPool2d → SphericalMaxPool2d

Every convolution reuses the exact source Parameter objects. Batch normalization, activation, dropout, and residual addition remain unchanged because they do not define a planar spatial neighbourhood. Final class scores use a cosine-latitude solid-angle average instead of ordinary pixel averaging.

At each output ray, kernel offsets lie in its local tangent basis, are carried to the unit sphere with the exponential map, and sample the ERP bilinearly with longitude wrapping. This operation is differentiable with respect to inputs and weights.

4. Run the tracked CC0 example

The repository includes a checksum-pinned, CC0 Poly Haven panorama for the documentation examples. Run one model per process:

python benchmarks/spherical_fcn_cam/run_experiment.py \
  --model resnet18 \
  --output-dir /private/tmp/panorai-spherical-fcn-cam \
  --preserve-input-resolution

Repeat with alexnet and vgg16. A missing model is downloaded automatically. Each model directory contains the input ERP, top-class CAM overlays, and a result.json containing:

• checkpoint URL, cache path, size, SHA-256 and cache status;

• canonical planar FCN parity error;

• input, feature and dense-logit shapes;

• every replaced layer and parameter-identity result;

• class scores and solid-angle CAM concentration;

• runtime, peak process memory, versions and input provenance.

For a custom ERP, provenance is mandatory:

python benchmarks/spherical_fcn_cam/run_experiment.py \
  --model resnet18 \
  --input /path/to/panorama.jpg \
  --input-license "dataset name and applicable terms" \
  --output-dir /private/tmp/my-spherical-cam \
  --preserve-input-resolution

The input must be a 2:1 ERP. Generated outputs should remain outside the repository unless their redistribution rights and provenance are separately established.

5. Why the dense map is smaller than the panorama

Fully convolutional means that the network accepts spatial dimensions other than its training crop. It does not mean that the output stride becomes one pixel.

ResNet18 reduces the input by a total factor of 32. Therefore:

1024×2048 ERP → 32×64 feature grid → 32×64 class grid
2048×4096 ERP → 64×128 feature grid → 64×128 class grid

VGG16 also produces a stride-32 terminal grid, followed by the valid 7×7 convolution converted from fc6:

1024×2048 ERP → 32×64 features → 26×58 class logits
2048×4096 ERP → 64×128 features → 58×122 class logits

CAM overlays are interpolated back to the ERP only for display. That interpolation does not add spatial evidence. A denser intrinsic map requires a smaller output stride, skip features, or repeated tangent-view evaluation.

6. Resolution is not angular scale

A 3×3 kernel on a 2048×4096 ERP covers a smaller angular neighbourhood than the same raster kernel on a 224×448 ERP. Running at native resolution therefore changes both the number of output positions and the angular context seen by every layer.

This does not imply that spherical weight transfer requires training. It means that a controlled experiment must declare:

• ERP resolution;

• angular spacing of kernel taps;

• effective receptive field;

• output stride;

• perspective FoV used by any tangent-view reference.

An explicit angular-scale policy or spherical scale pyramid should be tested before attributing scale differences to a need for fine-tuning.

7. Interpret a CAM correctly

The dense class channel is evidence used by an ImageNet classifier. A bright region means that it contributed strongly to a class score; it is not a calibrated per-pixel probability and does not necessarily cover the complete object.

Use spherical class-activation map rather than unqualified “saliency map.” In 360° literature, saliency often denotes predicted human gaze or fixation, which is a different task. ImageNet labels also do not directly match all Matterport3D or Stanford2D3D semantic classes.

8. Three controlled comparison scenarios

The current implementation supplies scenario 1. The proposed study keeps the same pretrained weights and compares:

1. Original spherical FCN, output stride 32. Preserve the classification topology and measure its native dense lattice.

2. Dilated spherical FCN, output stride 8. Remove late strides and replace them with spherical dilation so the receptive field is retained while the class lattice becomes denser.

3. Independent gnomonic oracle. Evaluate the unmodified perspective model on 224×224 tangent views at declared centers and FoV, then reproject its responses to the sphere.

A conventional planar FCN applied directly to the ERP is the distortion-unaware control. Compare maps before visualization upsampling using solid-angle-weighted error, correlation, top-fraction IoU, peak geodesic distance, class-rank agreement, rotation-equivariance error, runtime and peak memory. Repeat across multiple ERP resolutions while holding angular support fixed.

The current exponential-map neighbourhood is not yet proven numerically equivalent to a perspective gnomonic view. Scenario 3 is the independent oracle required to make or reject that claim.

9. Run the licensed public-dataset protocol

Given a locally prepared and licensed manifest, the batch runner selects at most one development panorama per spatial group, runs every model in a fresh process, and generates contact sheets:

python benchmarks/spherical_fcn_cam/run_public_datasets.py \
  --inputs /path/to/prepared-inputs.jsonl \
  --output-dir /private/tmp/panorai-spherical-fcn-public \
  --preserve-input-resolution

Matterport3D and Stanford2D3D are not distributed with PanorAi. Their bytes, derived visualizations and applicable terms remain outside the source and release artifacts. Current real-data results demonstrate executable localization, not semantic-segmentation accuracy.

10. Current limits

• No ImageNet or panoramic fine-tuning is performed.

• Batch-normalization statistics remain perspective-trained.

• Only AlexNet, VGG16, and ResNet18 have explicit FCN adapters.

• Native inference is CPU/reference code, not an optimized throughput claim.

• Unsupported image regions require explicit masks; black pixels are never inferred to be invalid.

• Exact gnomonic equivalence and output-stride-8 dilation remain future experimental scenarios, not current capabilities.

Continue with the implementation-oriented benchmarks/spherical_fcn_cam/README.md in the source checkout, the Tutorial: image processing on the sphere tutorial, and the exact PanorAi canonical spherical geometry coordinate contract.

11. High-capacity ImageNet control

InternImage-G is the optional high-capacity control. Its official checkpoint contains roughly three billion parameters and uses a 512x512 training crop. PanorAi preserves the learned parameters, replaces every DCNv3 sampling grid with a local east/north tangent grid on the ERP, ports ordinary spatial convolutions, and exposes a dense decomposition of the attention classifier. The spatial mean of that decomposition must reproduce the global logits.

Install the isolated optional dependencies with pip install "panorai[deep-learning-internimage]". The first inference downloads the pinned external snapshot only after explicit terms acceptance and verifies the full weight-shard sizes and SHA-256 values. No checkpoint is included in the PanorAi wheel or sdist.

This remains weak directional ImageNet evidence. It is not an object detector, and the native class lattice is smaller than the ERP because the backbone retains its learned stride.

12. One-object proxy-guided segmentation workflow

SphericalProxySegmenter encapsulates verified model acquisition, direct-spherical InternImage evidence, proxy conversion, streamed SAM 2.1 charts, same-instance expansion, solid-angle fusion, and cleanup:

Frozen ImageNet proxy vocabulary

ImageNet-1K has no direct industrial ontology for pipes, valves, pressure vessels, pumps, or process equipment. PanorAi therefore groups 15 pretrained ImageNet channels into eight geometric activation proxies. The table below is the complete public IMAGENET_PROXY_VOCABULARY; weights are fixed within each proxy concept.

PanorAi proxy concept	ImageNet classes (index, weight)	Possible industrial visual cue
circular-object	analog clock (409, 0.50); electric fan (545, 0.25); washer (897, 0.25)	circular instrument faces, flanges, or end caps
railing	bannister (421, 1.00)	guardrails and handrails
screen-like-object	fire screen (556, 1.00)	panels, screens, or framed planar structures
repeated-vertical-structure	organ (687, 1.00)	repeated upright pipes or tube banks
grid-like-structure	prison (743, 1.00)	grilles, cages, or repeated structural grids
elongated-object	bullet train (466, 1.00)	long equipment bodies or pipe-like structures
cylindrical-container	milk can (653, 0.35); steel drum (822, 0.25); barrel (427, 0.20); beaker (438, 0.20)	drums, tanks, or vessel-like containers
curved-structure	coil (506, 0.35); steel arch bridge (821, 0.35); reel (758, 0.30)	curved pipe runs, coils, or spools

These mappings mean only that a pretrained channel can provide a spatial cue for a visually related shape. They do not rename an ImageNet prediction as an industrial class, establish pipe/vessel accuracy, or replace an industrial detector or segmentation dataset. Every returned segment retains the exact proxy-class provenance that contributed to its evidence.

from panorai.experimental.deep_learning.segmentation import (
    SphericalProxySegmenter,
)

with SphericalProxySegmenter.from_pretrained(
    semantic_model="internimage-g",
    mask_model="sam2.1-hiera-large",
    device="mps",  # or "cpu"
    accept_upstream_terms=True,
) as segmenter:
    result = segmenter.predict(
        canonical_erp,
        projection="equirectangular",
        support_mask=canonical_support,
    )

The first prediction acquires missing files. InternImage is released before SAM is loaded, and SAM is released after segmentation, so both networks are not resident together. A later prediction reloads the verified local files without downloading them again.

The returned SphericalSegmentationResult contains segments, proposals, panoptic_map, unknown_segments, and complete diagnostics. Its default mask lattice preserves aspect ratio and is capped at 1024 rows; pass output_shape_hw=canonical_erp.shape[:2] for source-resolution masks at a substantially larger memory cost.

Use SphericalSemanticSegmenter directly when dense evidence or a compatible mask backend already exists; that lower-level class also supports an explicit class-agnostic mode with evidence=None. The convenience facade accepts canonical ERPs only. Camera-specific rasters must be converted outside the generic API. The output is automatic proxy-oriented segmentation, not ground truth, and insufficient evidence remains unknown.