Image-based modeling and rendering techniques greatly advanced the level of photorealism in computer graphics. They were originally proposed to accelerate rendering with the ability to vary viewpoint only. My work in this area focused on capturing and modeling real scenes for novel visual interactions such as varying lighting condition and scene configuration in addition to viewpoint. This work can lead to applications such as virtual navigation of a real scene, interaction with the scene, novel scene composition, interior lighting design, and augmented reality.
I will first briefly introduce my work on image-based modeling and rendering for varying viewpoint only. In the next part, I will present the inverse global illumination technique which refers to recovering reflectance models of various materials present in a real mutual illumination environment. The method’s input is a geometric model of the scene and a set of calibrated photographs taken with known light source positions. The result is a lighting-independent model of the scene’s geometry and reflectance properties, which can be rendered under novel lighting conditions using traditional graphics methods. The underlying philosophy is using low-parameter BRDF models and solving optimization problems to recover the parameters. Synthetic images rendered using recovered BRDF models are comparable to real photographs.
In the last part, I will present the techniques to extract an object-level representation of a real scene which can be rendered with modifications to the existing spatial configuration. The key components here involve automatic algorithms to segment the geometry from range images into distinct surfaces, and register texture from radiance images with the geometry. The top-down segmentation algorithm recursively partitions a point set into a binary tree with individual surfaces as leaves. Our image registration technique can automatically find the camera poses for arbitrary position and orientation relative to the geometry. I will demonstrate the effectiveness of these algorithms on real world scenes.