AI can now transform your eye reflections into intricate 3D renderings

A new study by scientists from the University of Maryland describes rendering these 3D reflections using an artificial intelligence (AI) visual rendering algorithm called neural radiance field or NeRF. 

Generating data for NeRF

A NeRF is a neural network that generates 3D representations of complex scenes. It takes multiple 2D images captured from different angles and synthesizes a continuous 3D view with high depth and detail.

The team of researchers started by taking multiple images using a high-resolution camera focused on an individual in motion. By examining the reflection in the person's eye, they could observe a mirror image of the field of view, which allowed them to identify objects in the surrounding area.

The images also captured details of the eye, such as the texture of the iris. To remove these details, the team employed texture decomposition. They trained a 2D texture map that learned the iris texture, which it then eliminated from the captured images. 

Computations were performed to monitor exactly where their eyes were looking using cornea geometry, which is about the same in all people. Additionally, this allowed the camera's angle to be determined and then map image coordinates onto the curved corneal surface. This information provided the viewing direction needed for the NeRF AI to reconstruct a 3D rendering of the surrounding.

Lights were placed out of the frame on the person's sides to shine a light on the objects in front. Multiple images were captured as the person moved around within the camera's view.

Using NerF AI to render 3D images

They employed a human eye to evaluate the NeRF AI, which resulted in a reasonable resolution rendition of the image with depth-mapped 3D rendering. However, synthetic tests using a fake eye before a digital image produced a clearer image with improved resolution in the 3D mapping.

For a third test, the team applied their method to eye reflection images from two music videos by Miley Cyrus and Lady Gaga to reconstruct what they were observing during filming.

The NeRF's rendering showed an LED light in Miley Cyrus's eye, which aligned with the video's concept of shedding a tear and being illuminated by a bright light. On the other hand, Lady Gaga's eye showed an unclear image. The team interpreted it as a camera on a tripod.

In the music videos, the musician was likely the main focus with well-lit surroundings. The lighting settings in everyday events, such as Zoom conversations or selfies, may be better suited for acquiring information about the surroundings.

You can check out more about their methodology and rendered images here.

The findings of their study are published in the pre-print server arXiv.

Study abstract:

The reflective nature of the human eye is an underappreciated source of information about what the world around us looks like. By imaging the eyes of a moving person, we can collect multiple views of a scene outside the camera's direct line of sight through the reflections in the eyes. In this paper, we reconstruct a 3D scene beyond the camera's line of sight using portrait images containing eye reflections. This task is challenging due to 1) the difficulty of accurately estimating eye poses and 2) the entangled appearance of the eye iris and the scene reflections. Our method jointly refines the cornea poses, the radiance field depicting the scene, and the observer's eye iris texture. We further propose a simple regularization prior on the iris texture pattern to improve reconstruction quality. Through various experiments on synthetic and real-world captures featuring people with varied eye colors, we demonstrate the feasibility of our approach to recover 3D scenes using eye reflections.