Robots level up: AI helps them understand material composition

The collaboration between MIT and Adobe Research paves the way for a future where robots possess a deeper understanding of materials, transforming industries and our interactions with technology.
Abdul-Rahman Oladimeji Bello
Ai Processor
Artificial Intelligence processor unit


Robots are getting more intelligent and more capable every day.

However, there's one challenge they still face: understanding the materials they interact with.

For example, imagine a robot in a car garage trying to pick up different items made of the same material. It would greatly benefit from knowing which items share the same composition, allowing it to apply the appropriate amount of force.

Identifying objects based on their material, known as material selection, has proven difficult for machines.

In addition, materials can appear different due to factors like object shape and lighting conditions, making it a complex problem.

However, MIT and Adobe Research researchers have made significant progress by harnessing the power of artificial intelligence (AI).

The team developed a groundbreaking technique that enables AI to identify all pixels in an image representing a specific material.

Even more impressive is that this method remains accurate even when objects have different shapes, sizes, and lighting conditions that may deceive human eyes.

Any of these factors doesn't trick the machine-learning model.

This breakthrough brings us closer to robots with a deeper understanding of the materials they interact with, enhancing their capabilities and precision.

The development of the model 

To train their model, the researchers used "synthetic" data—computer-generated images created by modifying 3D scenes to generate various ideas with different material appearances. Surprisingly, the developed system seamlessly works with natural indoor and outdoor settings, even those it has never encountered before.

Moreover, this technique isn't limited to images but can also be applied to videos.

For example, once a user identifies a pixel representing a specific material in the first frame, the model can subsequently identify objects made from the same material throughout the rest of the video.

The potential applications of this research are vast and exciting.

Beyond its benefits in scene understanding for robotics, this technique could enhance image editing tools, allowing for more precise manipulation of materials.

Additionally, it could be integrated into computational systems that deduce material parameters from images, opening up new possibilities in fields such as material science and design.

Robots level up: AI helps them understand material composition
Automation data analytic with robot

One intriguing application is material-based web recommendation systems. For example, imagine a shopper searching for clothing from a particular fabric.

By leveraging this technique, online platforms could provide tailored recommendations based on the desired material properties.

Prafull Sharma, an electrical engineering and computer science graduate student at MIT and the lead author of the research paper, emphasizes the importance of knowing the material with which robots interact.

Even though two objects may appear similar, they can possess different material properties.

Sharma explains that their method enables robots and AI systems to select all other pixels in an image made from the same material, empowering them to make informed decisions.

As AI advances, we can look forward to a future where robots are intelligent and perceptive of the materials they encounter.