AI + 3D Machine Vision Transforms Depalletization

ANDREA PUFFLEROVA AND MICHAL MALY, PHOTONEO

Manually offloading pallets requires scores of employees to work nonstop. Extensive job and safety training is required, and because the task is physically demanding, injuries often result, leading to increased absenteeism.

These are a few of the reasons companies are increasingly electing to automate depalletization. Robots can work around the clock every day of the year, without human intervention. And they require less floor space in which to operate.

Advanced AI algorithms can easily recognize randomly placed boxes of various sizes, shapes, materials, and textures. Courtesy of Photoneo.

Traditional approaches to automating pallet unloading include classic robot depalletization and delayerization. The former has its drawbacks: It is inflexible in cases of irregularities or unpredictable occurrences, such as handling damaged or randomly placed boxes. Similarly limited is delayerization, in which robots pick whole pallets that are laden with uniformly arranged boxes. This approach only works when boxes are of the same height.

A more advanced alternative is AI-powered depalletization, which combines artificial intelligence and powerful 3D machine vision to recognize various box types, pick them one by one, and place them in a predefined location. AI-powered depalletization is less expensive because the robot manipulates only one box at a time instead of the whole pallet. This option requires a smaller placement area than delayerization. For the same reason, the payload is lighter and the system requires a smaller and therefore cheaper robot and gripper.

As an illustration, consider 30- × 30-cm boxes, each weighing 20 kg. A standard pallet has a capacity of 12 boxes of this type (4 × 3 pieces). In this case, a smart depalletization system requires a robot with a payload of approximately 30 kg, including the gripper. In contrast, a delayerization system needs to manipulate the whole layer — meaning all 12 boxes or 240 kg at a time. The overall payload thus needs to be approximately 300 kg, depending on the robot type.

Advanced, AI-powered depalletization solutions are best suited for applications that require the unloading of boxes one by one from pallets. Such applications include e-commerce, fulfillment, and warehouse operations.

3D vision advancements

The process of smart depalletization starts with a 3D device that scans the whole pallet and sends the scan to a 3D-texture data set that is further processed by AI algorithms. Because this data is unaffected by external lighting conditions, it can be further used to flawlessly re-create the box shapes.

This form of 3D vision features high resolution acquired at fast scanning speed. Scanning volume is also critical. The scanning volume of the 3D vision system must cover the whole pallet and should be large enough to leave sufficient space for robotic manipulation. Considering these requirements, the 3D scanning device should generally be mounted approximately 3 m above the pallet, which corresponds to the minimum required scanning range.

The 3D vision system scans the whole pallet and sends the scan data to a 3D texture data set to be processed by AI algorithms. To ensure safe manipulation, the scanner is mounted at least 3 m above the pallet. AI in action (top). Courtesy of Photoneo.

If a depalletization application is located in a setting with challenging light conditions, such as in external light that is too bright, the 3D scanner can run algorithms for ambient light suppression. A noise filter can remove artifacts caused by the reflection of projected patterns and thus can significantly eliminate wrong measurements.

A crucial factor in automated depalletization is cycle time, which is affected by a number of elements, including scanning time. Each customer requires a solution that will provide the highest- quality data in the shortest time possible. And the shorter the scanning time, the faster the cycle time. Both these times can be significantly reduced by deploying a 3D vision system featuring a powerful and fast graphics processing unit (GPU) that reduces computation time — which is crucial to the overall scanning time. For instance, the NVIDIA Jetson TX2 4GB module enables users to run neural networks with double the compute performance and power efficiency of the Jetson TX1; this can allow a 60% faster computation time and a 200-ms shorter overall scanning time.

The deployed 3D vision must provide highest-quality data and precise measurements so that AI algorithms can further work with them. For an illustration, such data with measurements may involve about 3 million 3D points in every scan and an accuracy of 25 to 500 µm, depending on the scanning volume of the device. Best-in-class 3D vision is thus the first component needed for the successful recognition of individual boxes. The second is advanced AI algorithms.

AI overcomes challenges of tradition

Smart depalletization solutions are driven by sophisticated AI algorithms that constantly learn and immediately recognize all types of boxes. The best solutions are universal and immediately usable, without necessitating training of the system. When it comes across new types of boxes, it is able to retrain itself.

Some AI-powered depalletization systems are able to unload 1000 boxes in an hour. Courtesy of Photoneo.

Factors that pose major challenges to traditional solutions present no such issues for systems that combine AI and advanced 3D vision. These factors include various box sizes; irregular shapes, caused by damage, for instance; various colors, such as black; problematic surfaces that include shiny and reflective materials, varying textures, and protruding tapes; or even patterns and pictures that may trick the 3D vision.

Advanced depalletization systems can easily recognize boxes that are packed tightly one next to another. If the gap separating the individual boxes is too narrow — for example, 0.5 mm — weaker depalletization systems may confuse it with a line contouring the opening of only one box. This error does not happen with smart systems that use the most advanced method for segmenting the individual boxes on the basis of texture and 3D data — a convolutional neural network (CNN).

Pick and place

After successful recognition and localization, the system decides which box to pick and how to grasp it to maximize the suction power of the deployed gripper. Boxes placed on the top of the pallet come first. If the gripper fails to pick a box — due to a crinkled surface, for instance — the system sends feedback to the user, who initiates a corrective action. To ensure safe manipulation in the space between the pallet and the scanner, the system also needs to take into account the potential size of the box.

When the robot picks a box, the robot then places it in a predefined location, such as onto a conveyor belt. Here again, to ensure a safe placement, the system can either use the scan data to calculate the height of the box, or it can deploy an optical gate that is set to a few centimeters above the conveyor belt. When the box touches the optical beam, the robot drops it. This method ensures that even boxes of different sizes get dropped off from the same distance above the conveyor belt. Delayerization, in contrast, cannot handle boxes of various sizes.

The cycle time is dependent on several factors, including the already discussed scanning speed of the deployed 3D vision system; the environment, including the robot type; and the surface of the boxes and their contents. If a faster cycle time is required, the customer can deploy a multizone gripper that can pick several boxes of the same height at a time. In this case, it is important to consider the size limits of the placement area and not exceed it.

AI-powered depalletization

Automating depalletization by deploying a smart system increases productivity, throughput, and efficiency, reduces the required floor space, and saves operational costs. Employees can be relocated to tasks that require critical thinking, decision-making, or creativity. This, in turn, increases employee motivation and minimizes the danger of fatigue and errors.

The developments in AI and 3D machine vision dictate the direction of future technologies — among them automation, which improves processes, boosts productivity, saves time and costs, unburdens workers of monotonous and dangerous tasks, and positions a company against ever-stronger competition and market demands.

No less important is the fact that automating the depalletization process significantly reduces the risk of injuries and the detriments to long-term health. Manual operation involves the manipulation of heavy boxes that are often located a few meters above the ground. A robot, however, can handle weights of a few dozen kilograms and can work nonstop without getting tired.

Photoneo offers an advanced depalletization system that combines 3D vision with AI. The system was trained on a data set of more than 5000 types of boxes, enabling the AI algorithms to recognize and localize new types immediately. The result is that the system can unload 1000 boxes in one hour with 99.7% pick rate accuracy and a gripping precision of ±3 mm. The specially designed gripper can handle boxes of up to 50 kg.

Besides the powerful and reliable 3D vision and advanced AI algorithms, additional factors are involved in selecting an automated depalletization solution. One is the deployment and integration time. Every customer wants to save time and looks for a solution that will be convenient to use. Therefore, a short deployment and integration time plays an important role here. Another necessity is the solution’s universality, flexibility, and compatibility with a wide range of robotic brands.

Advanced automation

The most advanced automation solutions are based on AI; however, AI is only part of the whole system. To be able to meet the specific needs of a customer, systems integrators should have knowledge of the various types of grippers and their functions, and how to distribute the individual mechanical components — such as robots and 3D vision systems — deployed in an application. However, integrators need no expertise in the various robotic brands or in complicated 3D-related calculations, which are performed by the AI algorithms.

The developments in AI and 3D machine vision dictate the direction of future technologies — among them automation, which improves processes, boosts productivity, saves time and costs, unburdens workers of monotonous and dangerous tasks, and positions a company against ever-stronger competition and market demands. The same is true for logistics companies: Only those that adopt the most advanced automation technologies will be able to respond to the rising market expectations and to the urgent need for fast order fulfillment and accurate supply chain processes.

Meet the authors

Andrea Pufflerova is a public relations specialist at Photoneo, a developer of AI-powered robotic intelligence and industrial 3D vision. She has a master’s degree from the University of Vienna; email: marketing@photoneo.com.

Michal Maly, Ph.D., is co-founder and director of AI at Photoneo. He received his doctorate in computer science and artificial intelligence at the Faculty of Mathematics, Physics and Informatics at Comenius University in Bratislava (FMFI UK) in 2013. Maly is an expert in the construction of re-inforcement learning agents based on rationality. He also worked in computer security and published an innovative solution for secure distributed computing. His vision is to improve computer understanding of the 3D environment and enhance comprehension of real-world objects; email: sales@photoneo.com.