[nicolaihaeni]

Problem

With camera_model="fisheye" and 3DGUT (with_ut=True, with_eval3d=True), pixels whose ray is more than ~80° off
the optical axis render empty (alpha 0) for many real ~180° fisheye lenses. They also get no gradient, so they are
never trained.

Synthetic shell of Gaussians around the camera, OpenCV fisheye camera with the coefficients of a real ~180° lens,
k = [0.0056, −0.0405, 0.0273, −0.0055], 1224×1024, f ≈ 482 px. Left: current main / 1.5.3, everything outside the
dashed 80° circle is empty. Right: with this PR.

Cause

OpenCVFisheyeCameraModel::image_point_to_camera_ray (gsplat/cuda/include/Cameras.cuh) inverts the forward
polynomial θ_d(θ) = θ(1 + k1θ² + k2θ⁴ + k3θ⁶ + k4θ⁸) with Newton iterations started from
approx_backward_poly = {0, max_angle / max_normalized_dist}, a linear guess that the code itself calls very crude.
For these coefficients max_angle = 95.1° (capped by the image diagonal) and max_normalized_dist = 1.27, so the
guess overshoots the true angle by ~30 % near the edge. Beyond ~80° the start lies past the forward polynomial's
maximum (~103°), Newton converges to the second, non-physical root (~116°), and theta >= max_angle marks the ray
invalid. The cut-off depends on the coefficients and on the image size (it moves with max_normalized_dist): 80.0°
for the lens above, 84.2° for the same lens refined by COLMAP, 84.7° for the first lens on a 1400×1400 image. The
PyTorch reference _OpenCVFisheyeCameraModel uses the same start and has the same behaviour.

The ZipNeRF fisheye scenes used in examples/benchmarks/3dgut/ are not affected: their coefficients are positive
and small, so Newton finds the right root everywhere in the image (out to ~88° in the corners). The existing camera
tests draw coefficients as 0.01 * randn, which doesn't reach this regime either.

Reproduce (run once on main and once on this branch)

"""Render a synthetic checkerboard shell of Gaussians through an OpenCV-fisheye camera with whatever gsplat is
importable, and save the RGB image: python render_shell.py <out.png>"""
import math, sys
import numpy as np, torch
from PIL import Image
import gsplat
from gsplat import rasterization

torch.manual_seed(0)
n = 1_200_000
d = torch.randn(n, 3); d = d / d.norm(dim=1, keepdim=True)
lon = torch.atan2(d[:, 0], d[:, 2]); lat = torch.asin(d[:, 1].clamp(-1, 1))
cell = ((lon / math.radians(10)).floor() + (lat / math.radians(10)).floor()) % 2
col = torch.where(cell[:, None] > 0, torch.tensor([0.95, 0.75, 0.2]), torch.tensor([0.15, 0.35, 0.75]))
means = (d * 3.0).cuda(); scales = torch.full((n, 3), 0.012).cuda()
quats = torch.tensor([[1.0, 0, 0, 0]]).repeat(n, 1).cuda()
W, H, fx, fy, cx, cy = 1224, 1024, 481.6, 481.8, 612.75, 510.0
K = torch.tensor([[[fx, 0, cx], [0, fy, cy], [0, 0, 1.0]]]).cuda()
k = torch.tensor([[0.005581, -0.04054, 0.027259, -0.005462]]).cuda()  # a real ~180 deg fisheye lens
img, alpha, _ = rasterization(means, quats, scales, torch.ones(n).cuda(), col.cuda(), torch.eye(4)[None].cuda(), K,
                              W, H, camera_model="fisheye", with_ut=True, with_eval3d=True, radial_coeffs=k,
                              packed=False)
Image.fromarray((img[0].clamp(0, 1) * 255).byte().cpu().numpy()).save(sys.argv[1])
print(gsplat.__file__, "mean alpha", float(alpha.mean()))

Fix

Start Newton from the equidistant model, θ0 = θ_d (approx_backward_poly = {0, 1}), in CUDA and in the PyTorch
reference. For fisheye lenses θ_d/θ stays close to 1 (0.97–0.98 at 80–85° for the lenses above), so Newton converges
to the right root; in a float32 simulation of the CUDA code the error is < 1e-4° for all angles up to 95°.

Tests

• New test_fisheye_unprojection_valid_up_to_lens_edge (CUDA and reference, two real lenses): projects rays at 1–85°
  along the image diagonals and checks that image_point_to_camera_ray returns them as valid rays at the same angle.
  Without the fix it fails with "44 of 676 in-image points rejected; first rejected angle 80.0 deg".
• tests/test_cameras.py: 134 passed (CUDA 13.0, PyTorch 2.9.1, RTX 5090, BUILD_CAMERA_WRAPPERS=1).
• clang-format 22.1.5 reports no changes for Cameras.cuh.