This new AI technique may change how we see the world

MIT researchers developed a technique that captures images of an object from various angles and converts its surface into a virtual sensor that captures reflections.
Abdul-Rahman Oladimeji Bello
Glossy objects
Glossy objects

MIT News 

Imagine seeing around corners or peeking beyond obstacles that once blocked your view. It's like having X-ray vision. It might now be possible thanks to new research. 

A group of brilliant researchers from MIT and Rice University unveiled an incredible computer vision technology that can change how we see the world. We could soon live in a world where shiny objects become extraordinary "cameras" that let us view our surroundings through their unique lenses.

Their developed technique operates by capturing images of an object from various angles. Then, converting its surface into a virtual sensor that captures reflections. The artificial intelligence system then maps these reflections, enabling it to estimate depth within the scene and unveil novel perspectives that were once hidden from view.  

"We have demonstrated that any surface can be converted into a sensor using this formulation that transforms objects into virtual pixels and sensors. This technique holds enormous potential across multiple domains," explains Kushagra Tiwary, co-lead author of the research and a graduate student in the Camera Culture Group at the Media Lab.

The researchers, including co-lead author Akshat Dave from Rice University, Nikhil Behari, Tzofi Klinghoffer, Ashok Veeraraghavan, and senior author Ramesh Raskar, are set to present their groundbreaking findings at the prestigious Conference on Computer Vision and Pattern Recognition.

Are there any practical ways to use this technology?

The applications of this breakthrough are virtually limitless, as it empowers users to see around corners and beyond obstructions that traditionally obstructed their line of sight. One area where this technology could be truly transformative is autonomous vehicles. 

Imagine a self-driving car utilizing reflections from lamp posts or buildings it passes to understand its surroundings, even when obstructed by a parked truck. It's not just futuristic; it's practical and could revolutionize the safety and navigation of autonomous transportation. 

This new AI technique may change how we see the world
Big silver chrome ball

But there are challenges because real life isn't as simple as crime shows, where detectives magically enhance blurry reflections to solve crimes. Reflections can be distorted and offer a limited view of the world. How did the researchers overcome the challenges posed by reflections? They developed a three-step technique called ORCa (Objects as Radiance-Field Cameras).

First, they captured multiple images of an object from different angles, capturing various reflections on its glossy surface. Then, they transformed the object's surface into a virtual sensor that captures light and reflections. And that was it; the shiny object is now a "camera." 

Finally, the system uses the virtual pixels on the object's surface to recreate a 3D environment from the object's perspective.

ORCa's superpowers don't stop there. Analyzing multiple reflections can estimate the depth between the shiny object and other elements in the scene. It even predicts the object's shape. Also, thanks to its 5D radiance field representation of the scene, it can reveal hidden features once concealed by corners or obstructions. It's like seeing the world with a whole new set of eyes!

The researchers are now exploring applications in drone imaging and further enhancing the system's capabilities by incorporating additional visual cues. This groundbreaking technique is poised to redefine computer vision and unlock new dimensions of perception, expanding our understanding of the world around us.


Add Interesting Engineering to your Google News feed.
Add Interesting Engineering to your Google News feed.
message circleSHOW COMMENT (1)chevron
Job Board