# Tutorial: pretrained spherical FCN and class-activation maps | Field | Contract | | --- | --- | | **Application** | Localize semantic regions | | **Input** | One ERP and a frozen pretrained classifier/channel | | **Output** | Dense directional CAM evidence and optional FOV proposal | | **Strategy** | Sphere-native learned evidence | | **API** | `panorai.experimental.deep_learning` | | **Stability** | Experimental | | **Dependencies** | `panorai[deep-learning]`; external verified weights | | **Evidence** | Real CC0 output; CAM is not a mask or calibrated probability | ## How this application is composed The port replaces learned planar spatial layers with `panorai.core.torch` spherical convolution/pooling while retaining ordinary pointwise layers and the pretrained parameter objects. Solid-angle reduction produces global scores; retaining the dense class lattice produces CAM evidence. The model loaders and proxy facade are wrappers around that composition because artifact identity, layer conversion and provenance must be checked together. This tutorial ports three Torchvision ImageNet classifiers to differentiable spherical inference while preserving their learned parameters. The API is **Experimental** and opt-in under `panorai.experimental.deep_learning`; it is not part of the stable projection contract. See the {ref}`capability-map-image-processing` theme for the distinction between sphere-native sampling and planar models evaluated through projected views. **PanorAi-specific:** full spatial-layer inventory, exact parameter reuse, longitude-wrapped tangent sampling, spherical-area score pooling, checkpoint provenance, and explicit separation between CAM localization and semantic segmentation. ## Industrial application card: directional evidence, not segmentation ![Boiler Room ResNet18 prison CAM beside a registered multiview association diagram](../../_static/tutorials/application-semantics-boiler-room.jpg) This real output uses the public CC0 [Boiler Room](https://polyhaven.com/a/boiler_room) ERP at 512×1024. The exact ImageNet class `prison` (index 743), declared beforehand as the `grid-like-structure` proxy, ranked first with classifier probability 0.1920. The left panel shows a within-channel blue–yellow–red CAM with its colorbar. The right panel is explicitly a **method diagram**: independent region/FOV proposals in registered views can support one shared spatial hypothesis. It is not a measured second-view match. The native evidence lattice is only 16×32; interpolation is display-only. This is a deliberately readable channel, not cherry-picked proof of object recognition. The complete proxy-channel audit—including weak and failed channels—is documented in the industrial comparison and benchmark commands below. The component and FOV are region hypotheses, not a pipe/panel segmentation, ground truth, or calibrated pixel probability. Source, model, port, and derivative hashes are in {download}`image provenance <../../_static/tutorials/ATTRIBUTION.md>` and `application-gallery-metadata.json`. ## Cross-model regression comparison on the forest fixture The following outputs use the tracked CC0 Nature Reserve Forest ERP at its native `512×1024` resolution. All three backbones generated a `16×32` dense lattice. The overlays interpolate that lattice back to the ERP for inspection; interpolation does not create additional spatial evidence. Red means larger min-max-normalized evidence within that channel, not per-pixel probability. ### ImageNet ResNet18: fixed class `lakeside` ![ImageNet ResNet18 spherical CAM for lakeside](../../_static/tutorials/spherical-cam-imagenet-lakeside.jpg) This is a classical fixed ImageNet class channel. It demonstrates the object-oriented closed-vocabulary path and ranked third on this panorama; the map is weak evidence, not a claim that the panorama contains a labelled lake. ### Places365 ResNet18: fixed scene `forest/broadleaf` ![Places365 ResNet18 spherical CAM for forest broadleaf](../../_static/tutorials/spherical-cam-places365-forest.jpg) The Places365 scene channel concentrates on several vegetation masses. This is the closed-vocabulary scene extension, using the same frozen-weight spherical portability strategy. ### OpenCLIP RN50: prompt `a photo of a path` ![OpenCLIP RN50 spherical similarity for a photo of a path](../../_static/tutorials/spherical-cam-openclip-path.jpg) The prompt-defined channel shifts its stronger responses toward the ground. It is local image-text similarity from the dense reinterpretation, not the original global CLIP score and not a segmentation mask. ## 1. Install the optional stack Install Torch and Torchvision only when this experiment is required: ```bash pip install "panorai[deep-learning]" ``` The ordinary `panorai`, `panorai.geometry`, and `panorai.experimental` imports remain independent of Torchvision. Importing the explicit deep-learning submodule requires Torch, while Torchvision is loaded lazily when a pretrained model is requested. Supported official `DEFAULT` weights are: | architecture | perspective crop | FCN head conversion | | --- | ---: | --- | | AlexNet | `224×224` | first linear layer → valid `6×6` convolution | | VGG16 | `224×224` | first linear layer → valid `7×7` convolution | | ResNet18 | `224×224` | final linear layer → `1×1` convolution | ## 2. Download automatically or prefetch No checkpoint is stored in the repository or PanorAi distribution. The first inference request downloads a missing checkpoint through Torchvision into the user-controlled Torch cache. PanorAi verifies the URL hash prefix and records the complete SHA-256, path, size, URL, and whether the file was already cached. Prefetch all three models before an offline run: ```bash python benchmarks/spherical_fcn_cam/download_models.py ``` Or prefetch a subset: ```bash python benchmarks/spherical_fcn_cam/download_models.py \ --model resnet18 --model vgg16 ``` The command emits JSON with schema `panorai-pretrained-imagenet-cache/v1`. A cached file whose SHA-256 no longer matches the hash prefix in the official URL is rejected rather than silently used or deleted. Remove or replace such a file deliberately before retrying. The same behavior is available from Python through `prefetch_imagenet_weights()` and `load_pretrained_imagenet_model()`. These helpers contact the network only when their explicit model-loading operation finds a missing checkpoint. ```python from panorai.experimental.deep_learning import prefetch_imagenet_weights records = prefetch_imagenet_weights(("alexnet", "vgg16", "resnet18")) for record in records: print(record.model_name, record.cache_path, record.sha256) ``` Checkpoint licenses and terms remain those of their upstream publishers. The prefetch command is acquisition by the user, not redistribution by PanorAi. ## 3. What is ported ![Planar CNN spatial layers converted to direct spherical ERP inference](../../_static/tutorials/spherical-cnn-porting.svg) For AlexNet and VGG16, `ImageNetFCN` reshapes each linear classifier parameter into the corresponding convolution parameter. For ResNet18, the final linear classifier becomes a `1×1` convolution and the global average is moved after dense prediction. `port_module_with_report()` then recursively performs: ```text Conv2d → SphericalConv2d MaxPool2d → SphericalMaxPool2d ``` Every convolution reuses the exact source `Parameter` objects. Batch normalization, activation, dropout, and residual addition remain unchanged because they do not define a planar spatial neighbourhood. Final class scores use a cosine-latitude solid-angle average instead of ordinary pixel averaging. At each output ray, kernel offsets lie in its local tangent basis, are carried to the unit sphere with the exponential map, and sample the ERP bilinearly with longitude wrapping. This operation is differentiable with respect to inputs and weights. ## 4. Run the tracked CC0 example The repository includes a checksum-pinned, CC0 Poly Haven panorama for the documentation examples. Run one model per process: ```bash python benchmarks/spherical_fcn_cam/run_experiment.py \ --model resnet18 \ --output-dir /private/tmp/panorai-spherical-fcn-cam \ --preserve-input-resolution ``` Repeat with `alexnet` and `vgg16`. A missing model is downloaded automatically. Each model directory contains the input ERP, top-class CAM overlays, and a `result.json` containing: - checkpoint URL, cache path, size, SHA-256 and cache status; - canonical planar FCN parity error; - input, feature and dense-logit shapes; - every replaced layer and parameter-identity result; - class scores and solid-angle CAM concentration; - runtime, peak process memory, versions and input provenance. For a custom ERP, provenance is mandatory: ```bash python benchmarks/spherical_fcn_cam/run_experiment.py \ --model resnet18 \ --input /path/to/panorama.jpg \ --input-license "dataset name and applicable terms" \ --output-dir /private/tmp/my-spherical-cam \ --preserve-input-resolution ``` The input must be a `2:1` ERP. Generated outputs should remain outside the repository unless their redistribution rights and provenance are separately established. ## 5. Why the dense map is smaller than the panorama Fully convolutional means that the network accepts spatial dimensions other than its training crop. It does **not** mean that the output stride becomes one pixel. ResNet18 reduces the input by a total factor of 32. Therefore: ```text 1024×2048 ERP → 32×64 feature grid → 32×64 class grid 2048×4096 ERP → 64×128 feature grid → 64×128 class grid ``` VGG16 also produces a stride-32 terminal grid, followed by the valid `7×7` convolution converted from `fc6`: ```text 1024×2048 ERP → 32×64 features → 26×58 class logits 2048×4096 ERP → 64×128 features → 58×122 class logits ``` CAM overlays are interpolated back to the ERP only for display. That interpolation does not add spatial evidence. A denser intrinsic map requires a smaller output stride, skip features, or repeated tangent-view evaluation. ## 6. Resolution is not angular scale A `3×3` kernel on a `2048×4096` ERP covers a smaller angular neighbourhood than the same raster kernel on a `224×448` ERP. Running at native resolution therefore changes both the number of output positions and the angular context seen by every layer. This does not imply that spherical weight transfer requires training. It means that a controlled experiment must declare: - ERP resolution; - angular spacing of kernel taps; - effective receptive field; - output stride; - perspective FoV used by any tangent-view reference. An explicit angular-scale policy or spherical scale pyramid should be tested before attributing scale differences to a need for fine-tuning. ## 7. Interpret a CAM correctly The dense class channel is evidence used by an ImageNet classifier. A bright region means that it contributed strongly to a class score; it is not a calibrated per-pixel probability and does not necessarily cover the complete object. Use **spherical class-activation map** rather than unqualified “saliency map.” In 360° literature, saliency often denotes predicted human gaze or fixation, which is a different task. ImageNet labels also do not directly match all Matterport3D or Stanford2D3D semantic classes. ## 8. Three controlled comparison scenarios The current implementation supplies scenario 1. The proposed study keeps the same pretrained weights and compares: 1. **Original spherical FCN, output stride 32.** Preserve the classification topology and measure its native dense lattice. 2. **Dilated spherical FCN, output stride 8.** Remove late strides and replace them with spherical dilation so the receptive field is retained while the class lattice becomes denser. 3. **Independent gnomonic oracle.** Evaluate the unmodified perspective model on `224×224` tangent views at declared centers and FoV, then reproject its responses to the sphere. A conventional planar FCN applied directly to the ERP is the distortion-unaware control. Compare maps before visualization upsampling using solid-angle-weighted error, correlation, top-fraction IoU, peak geodesic distance, class-rank agreement, rotation-equivariance error, runtime and peak memory. Repeat across multiple ERP resolutions while holding angular support fixed. The current exponential-map neighbourhood is not yet proven numerically equivalent to a perspective gnomonic view. Scenario 3 is the independent oracle required to make or reject that claim. ## 9. Run the licensed public-dataset protocol Given a locally prepared and licensed manifest, the batch runner selects at most one development panorama per spatial group, runs every model in a fresh process, and generates contact sheets: ```bash python benchmarks/spherical_fcn_cam/run_public_datasets.py \ --inputs /path/to/prepared-inputs.jsonl \ --output-dir /private/tmp/panorai-spherical-fcn-public \ --preserve-input-resolution ``` Matterport3D and Stanford2D3D are not distributed with PanorAi. Their bytes, derived visualizations and applicable terms remain outside the source and release artifacts. Current real-data results demonstrate executable localization, not semantic-segmentation accuracy. ## 10. Current limits - No ImageNet or panoramic fine-tuning is performed. - Batch-normalization statistics remain perspective-trained. - Only AlexNet, VGG16, and ResNet18 have explicit FCN adapters. - Native inference is CPU/reference code, not an optimized throughput claim. - Unsupported image regions require explicit masks; black pixels are never inferred to be invalid. - Exact gnomonic equivalence and output-stride-8 dilation remain future experimental scenarios, not current capabilities. Continue with the implementation-oriented `benchmarks/spherical_fcn_cam/README.md` in the source checkout, the {doc}`spherical_image_processing` tutorial, and the exact {doc}`../../geometry-v1` coordinate contract. ## 11. High-capacity ImageNet control InternImage-G is the optional high-capacity control. Its official checkpoint contains roughly three billion parameters and uses a `512x512` training crop. PanorAi preserves the learned parameters, replaces every DCNv3 sampling grid with a local east/north tangent grid on the ERP, ports ordinary spatial convolutions, and exposes a dense decomposition of the attention classifier. The spatial mean of that decomposition must reproduce the global logits. Install the isolated optional dependencies with `pip install "panorai[deep-learning-internimage]"`. The first inference downloads the pinned external snapshot only after explicit terms acceptance and verifies the full weight-shard sizes and SHA-256 values. No checkpoint is included in the PanorAi wheel or sdist. This remains weak directional ImageNet evidence. It is not an object detector, and the native class lattice is smaller than the ERP because the backbone retains its learned stride. ## 12. One-object proxy-guided segmentation workflow `SphericalProxySegmenter` encapsulates verified model acquisition, direct-spherical InternImage evidence, proxy conversion, streamed SAM 2.1 charts, same-instance expansion, solid-angle fusion, and cleanup: ### Frozen ImageNet proxy vocabulary ImageNet-1K has no direct industrial ontology for pipes, valves, pressure vessels, pumps, or process equipment. PanorAi therefore groups 15 pretrained ImageNet channels into eight **geometric activation proxies**. The table below is the complete public `IMAGENET_PROXY_VOCABULARY`; weights are fixed within each proxy concept. | PanorAi proxy concept | ImageNet classes `(index, weight)` | Possible industrial visual cue | | --- | --- | --- | | `circular-object` | `analog clock` (409, 0.50); `electric fan` (545, 0.25); `washer` (897, 0.25) | circular instrument faces, flanges, or end caps | | `railing` | `bannister` (421, 1.00) | guardrails and handrails | | `screen-like-object` | `fire screen` (556, 1.00) | panels, screens, or framed planar structures | | `repeated-vertical-structure` | `organ` (687, 1.00) | repeated upright pipes or tube banks | | `grid-like-structure` | `prison` (743, 1.00) | grilles, cages, or repeated structural grids | | `elongated-object` | `bullet train` (466, 1.00) | long equipment bodies or pipe-like structures | | `cylindrical-container` | `milk can` (653, 0.35); `steel drum` (822, 0.25); `barrel` (427, 0.20); `beaker` (438, 0.20) | drums, tanks, or vessel-like containers | | `curved-structure` | `coil` (506, 0.35); `steel arch bridge` (821, 0.35); `reel` (758, 0.30) | curved pipe runs, coils, or spools | These mappings mean only that a pretrained channel can provide a spatial cue for a visually related shape. They do **not** rename an ImageNet prediction as an industrial class, establish pipe/vessel accuracy, or replace an industrial detector or segmentation dataset. Every returned segment retains the exact proxy-class provenance that contributed to its evidence. ```python from panorai.experimental.deep_learning.segmentation import ( SphericalProxySegmenter, ) with SphericalProxySegmenter.from_pretrained( semantic_model="internimage-g", mask_model="sam2.1-hiera-large", device="mps", # or "cpu" accept_upstream_terms=True, ) as segmenter: result = segmenter.predict( canonical_erp, projection="equirectangular", support_mask=canonical_support, ) ``` The first prediction acquires missing files. InternImage is released before SAM is loaded, and SAM is released after segmentation, so both networks are not resident together. A later prediction reloads the verified local files without downloading them again. The returned `SphericalSegmentationResult` contains `segments`, `proposals`, `panoptic_map`, `unknown_segments`, and complete diagnostics. Its default mask lattice preserves aspect ratio and is capped at 1024 rows; pass `output_shape_hw=canonical_erp.shape[:2]` for source-resolution masks at a substantially larger memory cost. Use `SphericalSemanticSegmenter` directly when dense evidence or a compatible mask backend already exists; that lower-level class also supports an explicit class-agnostic mode with `evidence=None`. The convenience facade accepts canonical ERPs only. Camera-specific rasters must be converted outside the generic API. The output is automatic proxy-oriented segmentation, not ground truth, and insufficient evidence remains `unknown`.