# Tutorial: image processing on the sphere

| Field | Contract |
| --- | --- |
| **Application** | Process directly on the sphere |
| **Input** | One ERP image |
| **Output** | Filtered, enhanced, transformed, edge, or pyramid ERP |
| **Strategy** | Sphere-native |
| **API** | `panorai.image_processing` |
| **Stability** | Experimental |
| **Dependencies** | Core; native acceleration optional |
| **Evidence** | Real CC0 output and analytic seam/pole tests |

## How this application is composed

`panorai.core.numpy` supplies fixed spherical convolution, gradients,
neighbourhood extrema and solid-angle reductions. `panorai.image_processing`
builds familiar Gaussian, Scharr, Canny, equalization and pyramid wrappers from
those primitives. Call a wrapper for one established operation; compose the
core modules directly when defining a new multi-stage method.

This tutorial applies familiar OpenCV-style stages directly to an
equirectangular panorama while keeping a constant angular footprint on the
sphere. The API is Experimental as `panorai-spherical-image-processing/v1`.

See the {ref}`capability-map-image-processing` theme in the spherical
computer-vision guide. These operators sample the ERP directly through
spherical rays; they do not project tiles to gnomonic space and blend them
back.

**PanorAi-specific:** constant-angle tangent neighbourhoods, longitude wrapping,
east/north derivatives, geodesic edge connectivity, ray-based transforms and
solid-angle histogram weighting.

The application card uses the public CC0
[Boma](https://polyhaven.com/a/boma) HDRI. It keeps one comparison legible at
README scale: constant-angle Gaussian smoothing above and geodesic Canny over
a dimmed source ERP below, both evaluated at 512×1024. The exact URL, authors,
checksum, parameters, and figure command are recorded in the downloadable
{download}`image provenance <../../_static/tutorials/ATTRIBUTION.md>`.

![Boma spherical Gaussian blur and geodesic Canny output](../../_static/tutorials/application-processing-boma.jpg)

The Nature Reserve Forest figures later in this tutorial remain tracked
regression/generalization fixtures produced by the older figure script.

## 1. Why a planar kernel is not spherical

In an ERP, longitude wraps at the left/right seam and the horizontal angular
scale represented by a raster neighbourhood changes with latitude. Applying a
planar kernel with ordinary image borders therefore creates a discontinuity at
the seam and gives the poles a different angular footprint from the equator.

PanorAi places a kernel in the local east/down tangent frame of every output
ray. Each tap is carried to the unit sphere with the exponential map, then
sampled from the ERP with longitude wrapping. The default tap spacing is one
vertical ERP pixel, or `180 / H` degrees.

![Spherical smoothing, tangent gradient, and geodesic Canny on a CC0 panorama](../../_static/tutorials/spherical-image-processing.jpg)

The Gaussian result above uses one angular kernel everywhere. The Scharr panel
shows the magnitude of derivatives in the local east/north frame. Canny uses
those directions for non-maximum suppression and uses spherical neighbours for
hysteresis, including across the longitude seam.

### Where each operation runs

The image-processing module does not materialize tangent images internally.
It evaluates tangent offsets directly on the sphere and samples the central
ERP. Gnomonic projection appears later in the feature pipeline, or explicitly
when an application chooses the separate `process_views` workflow.
The same classification, alongside projection, feature, geometry and SLAM
routes, is maintained in {doc}`spherical_capability_map`.

| Processing | Computation route | Spherical convolution? | Gnomonic projection? | What comes back |
| --- | --- | --- | --- | --- |
| custom `spherical_filter2d` | local tangent taps → exponential map → direct ERP sampling | yes; OpenCV-style correlation | no | filtered ERP |
| box blur | normalized box kernel through `spherical_filter2d` | yes | no | smoothed ERP |
| Gaussian blur | Gaussian kernel through `spherical_filter2d` | yes | no | smoothed ERP |
| median blur | geodesic tangent neighbourhood sampled directly from ERP | no; nonlinear spherical neighbourhood | no | smoothed ERP |
| bilateral filter | geodesic spatial weights plus radiometric weights | no; nonlinear spherical neighbourhood | no | edge-preserving ERP |
| Sobel or Scharr | east/north derivative kernels through `spherical_filter2d` | yes | no | signed tangent derivatives |
| Laplacian | four-neighbour kernel through `spherical_filter2d` | yes | no | signed second derivative |
| Canny | spherical Gaussian and gradients, then geodesic suppression and hysteresis | partly; its linear stages use spherical convolution | no | binary ERP edge map |
| histogram equalization | direct ERP histogram weighted by latitude-band area | no | no | equalized ERP |
| SLIC superpixels | Lab colour plus great-circle ray distance, solid-angle centroid updates, and seam-periodic connectivity | no | no | integer ERP superpixel labels |
| rotation | inverse-map output rays through a 3×3 sphere rotation | no | no | rotated ERP |
| resize | sample the exact output ERP rays | no | no | resized ERP |
| Gaussian pyramid | spherical Gaussian followed by ray-based ERP resize | yes, before each reduction | no | tuple of ERP levels |
| Laplacian pyramid | spherical Gaussian pyramid plus ray-resized residuals | yes, in its Gaussian stages | no | tuple of ERP residual levels |
| SIFT, ORB, or AKAZE features | ERP → overlapping gnomonic views → planar OpenCV detector → panorama bearings | preprocessing may use it; detector does not | yes | keypoints/descriptors mapped to ERP coordinates and unit bearings; no reconstructed image |
| experimental spherical DoG + SIFT | spherical Gaussian/DoG extrema → one tangent patch per accepted bearing → OpenCV SIFT descriptor | yes, for the detector scale space | only one local patch per descriptor; detection does not use projected views | seam-aware keypoints, angular scales, 128-value SIFT descriptors, and unit bearings |
| explicit `process_views` workflow | ERP → chosen tangent/cubemap views → caller operation → back-project and blend | no; the caller operation runs in the projected view domain | yes | reconstructed ERP plus support |

## 2. Run a complete processing chain

The executable example below creates a low-contrast ERP, equalizes it, applies
a custom sharpening kernel and Gaussian smoothing, computes gradients and
Canny edges, rotates the sphere, and constructs a scale pyramid.

```{literalinclude} ../../../scripts/run_documentation_examples.py
:language: python
:start-after: DOCS_SPHERICAL_PROCESSING_START = None
:end-before: DOCS_SPHERICAL_PROCESSING_END = None
:dedent: 4
```

`spherical_filter2d` follows OpenCV's correlation convention: the supplied
kernel is not flipped. Compatible NumPy `float32` and `float64` HW/HWC arrays
use the first-party C++ implementation when available. Set `backend="numpy"`
for the reference path or `backend="native"` when native execution is a hard
requirement.

Use the operators according to the noise and structure in the data:

| Operator | Use it for | Main caution |
| --- | --- | --- |
| box or Gaussian | denoising before gradients or keypoints | smooths real detail |
| median | isolated salt-and-pepper noise | nonlinear and currently NumPy-only |
| bilateral | denoising while retaining intensity edges | slower and parameter-sensitive |
| custom `filter2d` | sharpening or domain-specific linear kernels | non-finite taps are not renormalized |

## 3. Equalize by spherical area

Near the poles, an ERP allocates many pixels to a small solid angle. A raw
pixel-count histogram therefore lets polar rows influence the mapping more
than the same surface area near the equator. Spherical equalization uses
`weight[y] = cos(pi / 2 - pi * (y + 0.5) / H)`,

which is proportional to the exact solid angle of that latitude band.

![Solid-angle histogram equalization and ERP row weights on a CC0 panorama](../../_static/tutorials/spherical-histogram-equalization.jpg)

The aggressive contrast change in this example makes the mapping visible; it
does not imply that equalization always improves matching. Validate keypoint
repeatability and geometric inlier rate on the deployment domain. Use
`area_weighted=False` when exact ordinary `cv2.equalizeHist` behavior is
required, and use `mask=` to exclude invalid samples from construction of the
lookup table.

### What about CLAHE?

PanorAi does not currently expose a first-party sphere-native CLAHE operator.
OpenCV CLAHE can be supplied as the planar callback in an explicit
`process_views` workflow, but that result depends on face layout, FOV, overlap,
and blending. It is therefore a projected-view preprocessing choice, not an
equivalent implementation of contextual equalization on the sphere.

A sphere-native design needs geodesic contextual regions that cross the ERP
seam, stable behavior near the poles, and interpolation between local mappings
without introducing tile boundaries. It must also be evaluated on downstream
keypoint repeatability, match precision, and pose—not only visual contrast.
Until that contract exists, the global solid-angle equalizer is the canonical
first-party option and projected-view CLAHE must be recorded explicitly in the
experiment configuration.

## 4. Use preprocessing before spherical keypoints

The Stable feature pipeline detects descriptors on gnomonic virtual cameras.
Preprocessing changes the central ERP first, then PanorAi projects the enhanced
signal and maps detections back to panorama-frame bearings. It does not make a
planar detector itself seam-aware. The Experimental
`SphericalDoGSIFTPipeline` described in {doc}`03_features_and_matching` instead
builds its DoG scale space directly on the sphere.

```{literalinclude} ../../../scripts/run_documentation_examples.py
:language: python
:start-after: DOCS_SPHERICAL_PREPROCESSING_START = None
:end-before: DOCS_SPHERICAL_PREPROCESSING_END = None
:dedent: 4
```

Keep the same preprocessing configuration for both images in a matching pair,
record it with the feature-pipeline parameters, and compare against the
unprocessed baseline. Histogram equalization can amplify sensor noise, while
excessive smoothing can remove the corners the detector needs.

## 5. Transformations and scale space

`spherical_rotate` takes a proper 3×3 active rotation in PanorAi's canonical
`+X` right, `+Y` up, `+Z` forward frame. Reverse mapping covers every output
pixel without an artificial border. `spherical_resize` samples the exact rays
of the new raster.

Gaussian pyramids smooth on the sphere before each reduction. Laplacian
pyramids retain the residual against the expanded coarser level and can be
reconstructed with `reconstruct_laplacian_pyramid`. This is the appropriate
starting point for future scale-aware spherical detectors; it is not yet a
claim of SIFT scale-space equivalence.

## 6. Continue the workflow

- Continue to {doc}`03_features_and_matching` for SIFT, ORB, AKAZE, and
  descriptor matching on spherical virtual cameras.
- Compare direct spherical filtering with the complete
  project/filter/reconstruct route in {doc}`../../development/benchmarks` and reproduce the
  timing and yaw-equivariance probe with
  `scripts/benchmark_image_processing_routes.py`.
- Use {doc}`../how-to/spherical_image_processing` for the compact API guide and
  current limitations.
- Return to {doc}`02_projection_foundations` for the exact ERP ray convention.
- Treat {doc}`../../reference/stability` as the authoritative stability boundary.