NeLu3D: Neural Inverse Structured Light without Explicitly Modeling the Projector

Abstract

We propose NeLU3D, a neural inverse structured light (SL) method without explicitly modeling the projector. Structured Light (SL) is a popular technique that delivers precise 3D shape acquisition across diverse materials and object sizes. SL requires, however, careful modeling of the set-up, including geometric and radiometric calibration of the camera-projector pair; without it, reconstruction quality degrades. We calibrate the camera-projector system using a neural network that maps 3D positions to a set of projected patterns. Then, with as few as four monochromatic images (or two RGB images), our approach uses differentiable volume rendering to fit a surface to match SL captures. We scan over 25 objects of different shapes and reflectances to demonstrate the feasibility and quality of our method in a handful of projector-camera set-ups, including an extremely low-cost projector and an analog projector with fixed RGB pattern. We also showcase sub-millimeter accuracy with suboptimal patterns, where previous methods recover noisy 3D surfaces.

NeLU3D Pipeline

Our goal is recover a surface from captured structured light images of a scene without explicitly modeling the light source. We describe below how we bypass the explicit calibration of the light source and then use inverse rendering to reconstruct a surface.

NeLU3D Calibration

We move a flat calibrationg target across a volume while emitting

k

structured light patterns onto it (

k \geq 3

, but we have results going all the way to

k = 11

). The flat target is equipped with ChArUCo markers, which we use to establish relative pose between the target and a calibrated camera. We shoot rays from the camera and train a color MLP to predict, given a 3D coordinate, the corresponding color intensity for

k

patterns.

NeLU3D Reconstruction

As described above, we train a color MLP that skips the light source calibration - the weights of the color MLP remain frozen during reconstruction. For reconstruction. we cast the problem as inverse rendering by fitting scene geometry, represented by an MLP mapping 3D points to a signed distance function (SDF), and scene appearance, represented by an MLP mapping 2D image coordinates to global illumination contributions. The two MLPs are optimized to explain the captured images via volumetric rendering. At the end, we extract the surface from the zero level set of the SDF.

Results

Some results using NeLU3D with as few as three monochromatic patterns.

Comparison with Other SL Methods

We NeLU3D with three other recent Structured Light methods: TurboSL, LookUp3D, and ZNCC.
We use low-rank (with top 50 singular values) LookUp3D, since it is helpful to denoise the calibrated lookup tables, but the reconstructed point clouds are still covered by noisy outliers. TurboSL works really well with optimal patterns (e.g. 3 A La Carte, 4 or more Micro Phase Shifting, and 11 Gray codes), but it struggles with suboptimal patterns (e.g. 3 Hilbert or 3 random patterns). NeLU3D, on the other hand, outputs accurate surfaces with suboptimal patterns and with hard-to-calibrate projectors, while previous methods output incorrect and noisy results without additional and careful calibrations.

Black Keyboard

Milled Pawn w/ cheap LCD projector

We purchased the VOPLLS Mini Beaner N3, a 50USD LCD projector with cheap optics. This cheap projector lens has strong vignetting and distortion and the projected image does not focus on a single plane; these issues typically make SL struggle, but NeLU3D maintains sub-mm accuracy in these scenarios. Depth error for this figure is measured against ground truth CAD model.

Acknowledgments

This work was partially supported by NSF grants OAC-2411349 and OAC-2411221 and the NYU IT High Performance Computing resources, services, and staff expertise. We also want to thank Parsa Mirdehghan and Computational Imaging group at the University of Toronto for their work on Structured Light, including but not limited to TurboSL and the A La Carte patterns.

BibTeX


      @inproceedings{pereira2026nelu3d,
        title={NeLU3D: Neural Inverse Structured Light Without Explicitly Modeling the Projector},
        author={Pereira, Giancarlo and Fouhey, David and Silva, Claudio T and Panozzo, Daniele},
        booktitle={European Conference on Computer Vision},
        pages={565--583},
        year={2026},
        organization={Springer}
      }

NeLU3D: Neural Inverse Structured Light without Modeling the Projector

Neural LookUp3D (NeLU3D) is a neural inverse structured light scanning method without explicit projector modeling.