This new AI technique may change how we see the world

"We have demonstrated that any surface can be converted into a sensor using this formulation that transforms objects into virtual pixels and sensors. This technique holds enormous potential across multiple domains," explains Kushagra Tiwary, co-lead author of the research and a graduate student in the Camera Culture Group at the Media Lab.

The researchers, including co-lead author Akshat Dave from Rice University, Nikhil Behari, Tzofi Klinghoffer, Ashok Veeraraghavan, and senior author Ramesh Raskar, are set to present their groundbreaking findings at the prestigious Conference on Computer Vision and Pattern Recognition.

Are there any practical ways to use this technology?

The applications of this breakthrough are virtually limitless, as it empowers users to see around corners and beyond obstructions that traditionally obstructed their line of sight. One area where this technology could be truly transformative is autonomous vehicles. 

Imagine a self-driving car utilizing reflections from lamp posts or buildings it passes to understand its surroundings, even when obstructed by a parked truck. It's not just futuristic; it's practical and could revolutionize the safety and navigation of autonomous transportation. 

But there are challenges because real life isn't as simple as crime shows, where detectives magically enhance blurry reflections to solve crimes. Reflections can be distorted and offer a limited view of the world. How did the researchers overcome the challenges posed by reflections? They developed a three-step technique called ORCa (Objects as Radiance-Field Cameras).

First, they captured multiple images of an object from different angles, capturing various reflections on its glossy surface. Then, they transformed the object's surface into a virtual sensor that captures light and reflections. And that was it; the shiny object is now a "camera." 

Finally, the system uses the virtual pixels on the object's surface to recreate a 3D environment from the object's perspective.

ORCa's superpowers don't stop there. Analyzing multiple reflections can estimate the depth between the shiny object and other elements in the scene. It even predicts the object's shape. Also, thanks to its 5D radiance field representation of the scene, it can reveal hidden features once concealed by corners or obstructions. It's like seeing the world with a whole new set of eyes!

The researchers are now exploring applications in drone imaging and further enhancing the system's capabilities by incorporating additional visual cues. This groundbreaking technique is poised to redefine computer vision and unlock new dimensions of perception, expanding our understanding of the world around us.