This robot chef learns to mimic salad recipes just by watching videos

In the first step, the robot watched a video of a human cooking one of the recipes. After being trained with it, the robot was allowed to identify the recipe being prepared and make it.

Surprisingly, the videos also assisted the robot in creating its cookbook. According to the official release, using all these training resources helped the robot develop a ninth recipe on its own. 

“We wanted to see whether we could train a robot chef to learn in the same incremental way that humans can – by identifying the ingredients and how they go together in the dish,” said Grzegorz Sochacki from Cambridge’s Department of Engineering, and the paper’s first author, in an official press statement. 

Use of the neural network for training

The team used publicly available neural network algorithms to program the robotic chef to pick up recipes. 

The robot was thus trained to recognize distinct objects, including fruits and vegetables. 

The robot chef was even able to identify other objects, like the human demonstrator's knife and arms, hands, and faces. 

It watched 16 videos in total, so the robot correctly identified the recipe 93 percent of the time.  

“It’s amazing how much nuance the robot was able to detect,” said Sochacki. “These recipes aren’t complex – they’re essentially chopped fruits and vegetables, but it was really effective at recognizing, for example, that two chopped apples and two chopped carrots are the same recipe as three chopped apples and three chopped carrots.”  

The experiment shows that video content can provide a valuable data source for training robots to produce automated food. In the future, robots may assist human chefs in preparing food in various hospitality places. 

The results have been published in the journal IEEE Access. 

Study abstract:

Robotic chefs are a promising technology that can bring sizeable health and economic benefits when deployed ubiquitously. This deployment is hindered by the costly process of programming the robots to cook specific dishes while humans learn from observation or freely available videos. In this paper, we propose an algorithm that incrementally adds recipes to the robot’s cookbook based on the visual observation of a human chef, enabling the easier and cheaper deployment of robotic chefs. A new recipe is added only if the current observation is substantially different than all recipes in the cookbook, which is decided by computing the similarity between the vectorizations of these two. The algorithm correctly recognizes known recipes in 93% of the demonstrations and successfully learned new recipes when shown, using off-the-shelf neural networks for computer vision. We show that videos and demonstrations are viable sources of data for robotic chef programming when extended to massive publicly available data sources like YouTube.