Ambient-robust Inverse Rendering using Active RGB-NIR Imaging

Overview

Active RGB-NIR imaging for stable inverse rendering

Existing inverse rendering methods can be sensitive to uncontrolled ambient lighting. This project uses imperceptible NIR flash illumination to obtain stable point-light shading, while RGB images preserve visible-spectrum appearance for downstream rendering.

The method combines multi-view RGB observations, active NIR flash images, and an RGB-NIR BRDF model to recover geometry, roughness, metallic parameters, diffuse albedo, and environment illumination.

Figure 1 teaser showing the mobile RGB-NIR imaging system, captured observations, reconstructed geometry and reflectance, and relighting results. — Figure 1: overview of acquisition, reconstruction, and relighting.

Pipeline

RGB-NIR inverse rendering pipeline

The pipeline proceeds in three stages: RGB-based geometry initialization, NIR flash inverse rendering for robust geometry and reflectance refinement, and RGB environment inverse rendering for visible diffuse albedo and illumination recovery.

Stage 1

RGB initialization

Initialize object geometry from multi-view RGB images under natural ambient lighting.

Stage 2

NIR flash rendering

Use flash-only NIR measurements to refine geometry and estimate NIR reflectance.

Stage 3

RGB environment rendering

Recover RGB diffuse albedo and ambient environment maps using the refined reconstruction.

Figure 4 pipeline showing the three-stage RGB-NIR inverse rendering process. — Figure 4: three-stage RGB-NIR inverse rendering pipeline.

System

Imaging system design

The acquisition platform combines a mobile base, robotic arm, pixel-aligned RGB-NIR camera, and synchronized NIR flash. The system scans objects from dense viewpoints while keeping NIR illumination largely invisible to human observers.

Dataset

Acquired RGB-NIR inverse rendering dataset

The dataset contains multi-view RGB-NIR image pairs with active NIR flash under multiple ambient illumination conditions, together with a synthetic counterpart rendered to match the acquisition setup.

Real-world 3D dataset gallery

Four objects and fourteen ambient scenes, shown as pixel-aligned RGB/NIR depth-warp views.

14 assets

Synthetic 3D dataset gallery

Three synthetic objects and four ambient scenes, using the same RGB/NIR 3D viewer structure.

12 assets

Results

Reconstruction results

The paper evaluates RGB-NIR inverse rendering through BRDF validation, comparisons with passive and active RGB baselines, ambient-robust reconstruction, real-object relighting, ablations, and stress tests across material and scene complexity.

BRDF model validation

The RGB-NIR reflectance model is validated against measured material responses, showing how the NIR channel constrains reflectance parameters that are ambiguous in RGB-only observations.

Figure 7: validation of the RGB-NIR BRDF model — Figure 7. Validation of the RGB-NIR BRDF model on synthetic and measured observations.

Comparison against inverse rendering baselines

Baseline comparisons cover passive RGB inverse rendering, active RGB-only imaging, and diffusion-based inverse rendering. The result panels compare rendering, albedo, roughness, normals, environment maps, and relighting quality.

Figure 8: comparison with passive RGB inverse rendering — Figure 8. Comparison with passive RGB inverse rendering methods on synthetic and real-world scenes.

Figure 9: comparison with active RGB inverse rendering — Figure 9. Comparison with an active RGB inverse rendering setup.

Figure 10: comparison with diffusion-based inverse rendering — Figure 10. Comparison with diffusion-based inverse rendering.

Ambient robustness across environments

The method is evaluated under changing environment illumination, including controlled real-world captures, synthetic environment maps, and outdoor lighting conditions.

Figure 11: ambient-robust reconstruction on the real-world dataset — Figure 11. Real-world ambient-robust reconstruction across multiple environment maps.

Figure 12: RGB diffuse albedo reconstruction across synthetic environment maps — Figure 12. RGB diffuse albedo reconstruction across synthetic environment maps.

Figure 13: reflectance reconstruction under outdoor illuminations — Figure 13. Reflectance reconstruction under real-world outdoor illuminations.

Real-world relighting

Relighting results visualize how the reconstructed objects respond to new illumination. The video shows point-light relighting quality on a real captured object.

Figure 14: relighting results for real-world objects — Figure 14. Relighting results for real-world objects under different light directions.

Point-light relighting video on a reconstructed real object.

Ablation: NIR flash inverse rendering

The ablation isolates the contribution of NIR flash observations, showing how the active NIR signal improves decomposition quality under ambient illumination.

Figure 15: impact of NIR flash inverse rendering — Figure 15. Impact of NIR flash inverse rendering on reflectance and relighting.

Material types and scene complexity

Additional experiments test diffuse, specular, metallic, and multi-object scenes to show how reconstruction behaves as material composition and geometry become more challenging.

Citation

BibTeX

@inproceedings{chung2026ambient,
  title     = {Ambient-robust Inverse Rendering using Active RGB-NIR Imaging},
  author    = {Chung, Hoon-Gyu and Kim, Jinnyeong and Kang, Hyunwoo and Baek, Seung-Hwan},
  booktitle = {SIGGRAPH Conference Papers},
  year      = {2026},
  doi       = {10.1145/3799902.3811078}
}