Advancements in 3D Vision, Software Bolster Bin Picking

ADRIAN KRATKY, PHOTONEO S.R.O.

Industrial robots are the key element in automating processes throughout the manufacturing sector, with their numbers rising globally each year. In the U.S., for example, their numbers reached 200 robots per 10,000 employees in 2017, ranking seventh in the world. The front-runner was Korea with 710, followed by Singapore with 658¹.

Numerous production processes are automated today that were inconceivable a few years ago. Of the many challenges to overcome, one of the primary ones is in vision-guided robotics, particularly in the area of bin picking, which involves the robotic selection and acquisition of objects at random from within a bin. What the robot does with each part after picking it depends on the specific application. Sometimes the drop-off position and orientation don't matter; the robot picks the part and simply drops it into another bin or on a conveyor belt, such as in depalletization applications.

A robot is able to localize and pick objects using a 3D vision system. Courtesy of Photoneo s.r.o.

In other cases, the robot places picked parts in an oriented manner. Such pick-and-place applications include automated assembly of parts or post-processing, and may involve inspection, which enables a part to be rejected after being picked. This maximizes the efficiency of the entire automated process.

Another way to categorize bin picking is based on the way individual parts are placed in the bin before being picked. From this perspective, bin picking can be divided into three main categories: structured, semistructured, and random.

Structured bin picking is the most simple and straightforward robotic application. Objects are placed in a bin in an ordered and predictable pattern so that the robot can pick them easily. Consider the scenario where bottles stand in a basket next to each other, with caps facing up to be easily picked by the robot. Even though placing the objects in a bin in such an organized manner requires extra time, the part-to-part cycle time is generally shorter.

Random bin picking is complicated to perform because the individual parts are placed in a container in a completely random and unpredictable pattern. Courtesy of Photoneo s.r.o.

Semistructured bin picking covers applications in which objects are placed in the bin with some predictability. It is thus more difficult for the robot to localize and pick the parts. A sample use case is the picking of parts that are placed in piles, such as metal plates used in a hot forming process. The part-to-part cycle time depends on the individual application and on the level of organization.

In structured bin picking, parts are ordered and placed in a predictable pattern. Courtesy of Photoneo s.r.o.

In the case of random bin picking, the position and orientation of the individual objects in a bin are completely random and unpredictable; for example, random bin picking is ideal in the case of a container full of randomly positioned small metal parts that need to be picked one by one. The parts may overlap or be entangled, which further complicates their localization and subsequent picking. This category is the most advanced and brings bin picking to new realms; it's also contingent on advancements in machine vision and gripper technologies.

Semistructured bin picking is more difficult for a robot because the position of the objects in a bin is only partly predictable. Courtesy of Photoneo s.r.o.

Getting a grip

Gripper design has a direct impact on whether bin picking will be successful. First, the tool should be designed as an ejector, be it a vacuum cup, magnet, or parallel fingers. The end effector should be compact to prevent collisions, and its functional mechanism should not exceed the contours of the object that is to be picked. It should be able to grasp the object regardless of how the part is oriented in the bin. Of significant help to picking objects that are positioned in the corners or close to the walls of the bin is a mechanism with six degrees of freedom. The second offset should be as close as possible to the end effector.

Another challenge is presented by localization of objects in the bin. Picking known objects is relatively easy; the robot performs the action using a vision system that identifies items of known geometry by comparing them to a CAD model. This "analytical" solution is the traditional option — widely used, but limited in terms of its nonuniversality and the fact that it serves a single purpose in a particular application. At the same time, this solution is robust and offers a repetitive approach that enables oriented placing.

A major problem in bin picking is caused by localization of unknown objects. Bin picking applications such as these do not use CAD-based localization. Instead, the system detects the location of the objects using machine learning. The robot can then localize and pick objects of various shapes and sizes. This machine learning solution can often be found in applications for e-commerce, warehouses, logistics, and processing of parts that vary in shape or size. It provides higher universality and faster setup times, though it is not appropriate for oriented placing and is thus limited to pick-and-drop applications.

The object material is another factor that influences localization. Rigid parts can be picked without much trouble, but picking of soft or pliable objects, such as plastic bags, is much more complicated to perform. It is also important to find a compromise between the level of precision, speed, and overall robustness. In the world of robotics and machine vision, the future development revolves around generic handling of nonrigid objects, shape correction, and advanced manipulation.

Path planning

In the context of the challenges of bin picking, it is important to consider safety, which is interconnected with path planning. This feature, as the name implies, enables the robot to plan the trajectory of its robotic arm to avoid obstacles and prevent collisions. Generally, there are three options. The first one presents full planning of joint and linear trajectories and full collision avoidance. This solution is optimal for bin picking applications that use deep bins and complex environments. The trade-off for reliable collision prevention is lower speed.

These advancements are only a sampling of what is coming. Step by step, the technology is changing lives and pushing the borders of the possible ever further.

A second option is planning of linear trajectories only; the robotic arm approaches the part to be picked in a linear way from above. The length of the end effector on the extender needs to equal the depth of the bin, making it suitable for shallow bins. The collision avoidance is <100%, the discrepancy of which is compensated for by higher speed and reduced computation complexity.

The third option is to turn off path planning completely. Because the risk of a collision is extremely high in this case, it is only suitable for applications where no obstacles are within reach of the robot — for example, shallow bins or flat surfaces such as conveyor belts. If these conditions are met, the customer gets very high speed, given the elimination of calculation time and reduction of the overall computation time.

Skills required

It may come as a surprise that when assessing the challenges of bin picking, the skills of robotic programmers are important as well. But there is a good reason to include them here. Most bin picking applications are developed on greenfield, from scratch rather than from an already existing program. This requires a wide set of skills, which are beyond the scope of the majority of robotic programmers. Their expertise in CAD and robotic programming usually does not extend to the other skills necessary for developing a bin picking application. These skills range from software engineering to mathematics, geometry, robotic kinematics, 3D vision, and localization. On top of all this, the required skills are often specific to a particular robotic brand.

Eyes of the robot

Bin picking has developed rapidly in recent years. The number of robots performing material handling to unburden human workers of manual tasks and increase efficiency is growing every year. However, robots were lacking something essential in the beginning: the eyes. Even for humans, it is impossible to determine the orientation of an object if its features cannot be seen. This is an important fact to mention considering that oriented placing plays a crucial role in the development of bin picking.

Picking of envelopes is a pick-and-drop application in which the drop-off position and orientation do not matter; the robot simply drops the part on a conveyor belt. Courtesy of Photoneo s.r.o.

High-level 3D vision systems are, therefore, an inextricable part of any advanced bin picking application. There are many products on the market that can capture a 3D image of an object. Such vision systems produce data for the calculation of the object's 3D coordinates, which are then used by the robot to localize the part and subsequently pick it. Each product is based on a different technology — be it time of flight, stereo vision, or structured light. Every one of these methods offers a different set of benefits but also imposes limitations. One of the biggest constraints is that most products are able to capture only static scenes. Such products are categorized as 3D scanners. 3D cameras, which enable 3D imaging of objects in motion, provide an output of significantly lower quality.

In pick-and-place applications, the robot picks an object and places it in an oriented manner into an exact position. Courtesy of Photoneo s.r.o.

The only method that enables the capture of scenes in rapid motion in high resolution and with high accuracy is a technology that was developed only a few years ago. It has been patented as the "parallel structured light" technology and is implemented by a custom CMOS image sensor that has a mosaic shutter. This method has given birth to a 3D camera with the highest resolution and accuracy for scanning in motion.

The development of this parallel structured light technology has been significant for the world of bin picking because it has filled the gap that traditional technologies have not been able to fill. Manufacturers can now automate bin picking procedures that involve moving objects, such as parts placed on a conveyor belt or other applications with moving scenes.

The combination of high-quality, cutting-edge 3D cameras with advanced bin picking software brings robot capabilities ever closer to those of humans. This is a crucial element in robot development because vision and intelligence are what make humans capable and successful. The ability to learn and make the right decisions was missing in the bin picking world; merging 3D machine vision with the complex algorithms of bin picking software has turned this once inconceivable idea into a reality. A material no longer needs to be precisely positioned in a specific place nor stacked in a feeder to be localized and picked by the robot. Robots have been given eyes and intelligence, enabling them to learn and make decisions. And by implementing machine learning into bin picking, we are closer than ever to fully automated and 100% reliable bin picking.

Real-time reconstruction

Despite recent advancements in bin picking, crucial developments lie ahead. One groundbreaking feature is the real-time reconstruction of the robot's surrounding environment. Achieved through understanding, segmentation, and classification, this reconstruction will enable the robot to react to situations and eliminate the risk of collision with the changing environment.

We can also expect great flexibility in object localization. In general, the user can choose between CAD-based localization, a machine learning-driven approach, or, alternatively, any other algorithm developed by a third party in the form of a pretrained network. It is also not too far-fetched to imagine an autonomous mobile robotic platform with a system that is able to recognize or detect a bin containing objects, or the position of the objects' oriented placing.

These advancements are only a sampling of what is coming. Step by step, the technology is changing lives and pushing the borders of the possible ever further.

Meet the author

Adrian Kratky is director of robot vision at Photoneo s.r.o. In 2008, he graduated from the Slovak University of Technology in Bratislava, Faculty of Electrical Engineering and Informatics, with a specialization in robotics. His team is working on an algorithm for CAD-based object recognition, which is one of the pillars of bin picking applications; email: sales@photoneo.com.

Reference

1. N. McCarthy (April 29, 2019). The countries with the highest density of robot workers, www.statista.com/chart/13645/the-countries-with-the-highest-density-of-robot-workers.