This Two-Armed Robot is the Most Nimble-Fingered Machine Ever Created

This robot uses machine learning to understand how to grasp unfamiliar objects.
Jessica Miley

The world’s most dexterous robot can recognize and handle complex shapes and sort materials with lightning efficiency. The low-key looking robot has been designed by Ken Goldberg a U.C. Berkeley professor and his graduate student Jeff Mahler.

The key to the nimble robot is the use of software called Dex-Net that allows the robot to pick up even odd shaped objects with speed. Professor Goldberg showed off the latest version of the robot in San Francisco at EmTech Digital, an event dedicated artificial intelligence hosted by MIT Technology Review.

The unassuming robot dexterity would have useful applications in hospitals and warehouses. Dex-Net learns to grasp by trying to pick up objects in a virtual environment.

Dex-Net trained for hours in simulation

This process trains a deep neural network through trial and error. While this method of learning is lengthy, it reaches an outcome where Dex-Net can recognize an object it has never seen before. It can even learn to tune an object over to examine it more carefully before deciding on the best way to grasp it.

The most recent physical version of the system consists of two robot arms and a high-resolution 3-D sensor. One arm is equipped with a gripper and the other with a suction cup, each arm is controlled by a different neural network.

New metric helps comparison

As each object is presented to the robot, it is scanned and then the software looks at both networks before making a quick decision whether to suck or grip the object in question. The world’s most dexterous robot got its title from a newly designed way to measure the robot picking success.

Called “mean picks per hour,” the metric is calculated by ‘multiplying the average time per pick and the average probability of success for a consistent set of objects.’ This new way of measuring dexterous robots means that labs developing these kinds of robots can more accurately share their results.

“We’ve been talking about how to align our results so that we see progress,” Goldberg says. “It all depends what robot you’re using, what sensor you’re using, and—very importantly—what objects you’re using.”

The best robots out there are currently capable of roughly 70 and 95 mean picks per hour. Humans capable of between 400 and 600 and Goldberg's new Dex-Net machine can achieve a pretty impressive 200 to 300 mean picks per hour.

Robots to overtake humans

Goldberg predicts that robots will match or beat humans in their pick ability within five years. grasping and moving unpredictable, awkward or unusual shaped objects has been a challenge for robotics engineers to achieve.


While many robots are fast and precise when doing repetitive tasks, they fail under unpredictable conditions. Robots like Dex-Net that can respond to changing and unstable environments will have far greater application than just in factories and warehouses.

Their ability to understand new objects may make them suitable for use in more personal environments such as aged care and hospitals. Professor Goldberg will present his findings again at a conference in Australia later this year.