MIT researchers combined traditional characteristics of computer vision with radio sensing to enable a robot to detect occluded, or blocked objects. In application, the development could streamline e-commerce fulfillment in warehouses or help a machine detect a particular object in a disordered context. It also directly addresses the notions of perception and picking, both of which are roadblocks in the industry today, said Alberto Rodriguez, associate professor in the Department of Mechanical Engineering. With optical vision alone, robots are unable to find items packed within boxes or hidden behind another object on a shelf. Radio waves, because they are able to pass through walls, are an attractive option in overcoming this obstacle. Radio identification, used in the tracking and detection of books and pets, consists of two main components: a reader and a tag. The reader emits a radio frequency (RF) signal, which is modulated by the tag, a tiny computer chip, and reflected back to the reader. The reflected signal gives information about the location and identity of the tagged item. Recognizing this utility, the researchers decided to apply the technology to robotics. “RF is such a different sensing modality than vision,” Rodriguez said. “It would be a mistake not to explore what RF can do.” The robot that the group developed, called RF Grasp, uses a camera and an RF reader in tandem to find and grab tagged objects, event when they’re fully blocked from camera’s view. The camera is situated on the wrist of the grasping hand attached to the robotic arm, with the RF reader standing independent of the robot. The reader relays tracking information to the robot’s control algorithm, which is constantly collecting both RF tracking data and a visual picture of its surrounding. “The robot has to decide, at each point in time, which of these streams is more important to think about,” said Tara Boroushaki, a research assistant in the Signal Kinetics Group at the MIT Media Lab. “It’s not just eye-hand coordination; it’s RF-eye-hand coordination. So the problem gets very complicated.” The process begins with the robot pinging the target object’s RF tag to approximate its location. “It starts by using RF to focus the attention of vision,” said Fadel Adib, associate professor in the MIT Media Lab. “Then you use vision to navigate fine maneuvers.” Compared with similar robots equipped with only a camera, RF Grasp was able to pinpoint and grab its object with about half as much total movement. The system also demonstrated the ability to “declutter” its environment by removing objects obstructing its target or constricting access. That ability, Rodriguez said, demonstrates RF Grasp’s “unfair advantage” over robots without penetrative RF sensing. “It has this guidance that other systems simply don’t have,” Rodriguez said. The robot could one day perform order fulfillment in packed e-commerce warehouses. The RF sensing could instantly identify an item without the need to manipulate the item to scan a barcode. “RF has the potential to improve some of those limitations in industry, especially in perception and localization,” Rodriguez said. The work will be presented at the IEEE International Conference on Robotics and Automation.