X-ray vision? MIT's latest headset achieves just that

MIT scientists have engineered an X-ray vision augmented reality headset that combines computer vision and wireless perception to automatically locate items that are hidden from view.

There is one catch though: the hidden items have to have been labeled with RFID tags.

The headset is called X-AR and in tests conducted in a warehouse-like environment, it could localize hidden items to within 9.8 centimeters, on average. It also confirmed that users picked up the correct item with 96 percent accuracy.

“Our whole goal with this project was to build an augmented reality system that allows you to see things that are invisible — things that are in boxes or around corners — and in doing so, it can guide you toward them and truly allow you to see the physical world in ways that were not possible before,” said in a press release Fadel Adib, who is an associate professor in the Department of Electrical Engineering and Computer Science, the director of the Signal Kinetics group in the Media Lab, and the senior author of a paper on X-AR.

To create this novel invention, the researchers first had to outfit an existing headset with an antenna that could communicate with RFID-tagged items.

“One big challenge was designing an antenna that would fit on the headset without covering any of the cameras or obstructing its operations. This matters a lot since we need to use all the specs on the visor,” said co-author and former postdoc Aline Eid, who is now an assistant professor at the University of Michigan.

Synthetic aperture radar

Once the team had built an effective antenna, they focused on using it to localize RFID-tagged items by leveraging a technique known as synthetic aperture radar (SAR).

SAR allows X-AR to utilize visual data from the headset’s self-tracking capability to build a map of the environment and determine its location within that space.

“While it presented a challenge when we were designing the system, we found in our experiments that it works well with natural human motion. Because humans move around a lot, it allows us to take measurements from lots of different locations and accurately localize an item,” said co-author Laura Dodds.

The headset offers menus from which a user can select an object from a database of tagged items. Once the object is spotted by the headset, it is surrounded by a transparent sphere so the user can see exactly where it is in the room. 

“We abstracted away all the technical aspects so we can provide a seamless, clear experience for the user, which would be especially important if someone were to put this on in a warehouse environment or a smart home,” concluded co-author Maisy Lam.

The study was published by MIT.

Study abstract:

We present the design, implementation, and evaluation of X-AR, an augmented reality (AR) system with non-line-of-sight perception. X-AR augments AR headsets with RF sensing to enable users to see things that are otherwise invisible to the human eye or state-of-the-art AR systems. Our design introduces three main innovations: the first is an AR-conformal antenna that tightly matches the shape of the AR headset visor while providing excellent radiation and bandwidth capabilities for RF sensing. The second is an RF-visual synthetic aperture localization algorithm that leverages natural human mobility to localize RF-tagged objects in line-of-sight and non-line-of-sight settings. Finally, the third is an RF-visual verification primitive that fuses RF and vision to deliver actionable tasks to end users such as picking verification. We built an end-to-end prototype of our design by integrating it into a Microsoft Hololens 2AR headset and evaluated it in line-of-sight and non-line-of-sight environments. Our results demonstrate that X-AR achieves decimeter-level RF localization (median of 9.8 cm) of fully-occluded items and can perform RFvisual picking verification with over 95% accuracy (FScore) when extracting RFID-tagged items. These results show that X-AR is successful in extending ARsystems to non-line-of-sight perception, with important implications for manufacturing, warehousing, and smart home applications