A novel CMOS technology opens up automation possibilities in logistics operations.
ANDREA PUFFLEROVA AND MARCEL ŠVEC, PHOTONEO
Logistics can spell the difference between success and failure in business. Receiving, storing, inspecting, sorting,
packaging, labeling, shipping
— all of these steps need to happen
quickly to meet rising customer demands and deadline expectations. Maximizing throughput and ensuring process safety and reliability by properly optimizing logistics operations is therefore indispensable.
Automation is a key means to achieve this. The most complex and sophisticated tasks require the deployment of 3D machine vision that can provide the precise data necessary for robot navigation in pick-and-place applications, object inspection, quality
control, dimensioning, and other logistics tasks. Advanced high-tech solutions based on machine vision allow logistics centers to boost efficiency, ensure positive customer experiences, and keep up with the competition.
Moving parcels are 3D-scanned and sorted into various containers. Courtesy of Photoneo.
Automation possibilities in the logistics sector are tightly linked to and dependent on innovations in machine vision. One of the major advancements of recent times is the ability to capture moving objects and get high-quality
3D data without motion artifacts. With this advancement, automating logistics
tasks is no longer limited to static scenes, and robots can recognize and handle objects that are in fast and constant motion.
3D scanning in fast motion
Not that long ago, high-resolution 3D area sensing was restricted to capturing perfectly stationary objects. If either the scanned object or the camera (or both) moved, the 3D image was degraded by motion artifacts. This limitation also defined applications that could be automated.
Among the most popular technologies deployed in logistics applications to navigate robots are structured light, time of flight, and active stereo vision.
The structured light approach owes its popularity to the high resolution and submillimeter accuracy that it offers. Where there is no movement, the technology can provide exceptional results. The problem arises when an application requires capturing the image of moving objects. Because a structured light system projects a sequence of light patterns onto a scene, if the scanned object moves, the projected light code gets broken. This results in a 3D image full of flying artifacts, high noise, and incomplete point clouds. Therefore, the structured light approach can only be used in static applications.
A custom-made vacuum gripper used by Dedoles ensures reliable picking of items in motion. Courtesy of Photoneo.
In practice this means that if a robot is to first recognize and then pick objects moving on a conveyor belt, the line needs to be stopped to allow a 3D scan acquisition of each single item. Obviously, this affects cycle times and throughput.
The time-of-flight (TOF) or active stereo systems have proved to be a better solution for dynamic applications. This is due to their high scanning speed, the standard being 30 fps.
These systems, however, cannot be used in applications that require high-quality and accurate 3D data. Their resolution is typically VGA (640 × 480 pixels), and their accuracy can only be determined in millimeters. The 3D image is rather noisy and missing fine details on the edges of the scanned objects. As a result, TOF systems are not suitable for vision-guided robotics that requires precise robot navigation
and delicate object handling. This means that TOF and active stereo
systems can be used to automate logistics applications with dynamic scenes, but only if the task does not
require high-quality 3D data and
accurate robot navigation.
Eliminating motion artifacts
As the speed of logistics increases, planners must contend with keeping pace without interrupting the operations flow. And to do so, they are turning to a method of 3D scanning that enables the capture of large areas in fast motion, without artifacts.
A single sweep of laser light is projected onto the scene and then produces structured light patterns on the sensor side by a unique pixel modulation in parallel. This means that although the laser light emitted from the projector is on the entire time, the individual sensor pixels are repeatedly turned on and off during one exposure window, which allows the sampling of individually coded light patterns at one point in time and in one frame — hence, the name “parallel structured light” technology. This is an opposite approach to structured light technology, which is based on a sequential projection of structured light patterns.
A robot performs precise recognition and picking of items hanging on a moving overhead conveyor, where they may swing and rotate. Courtesy of Photoneo.
The ability to scan fast-moving objects in 3D in high quality without the need to interrupt the objects’ movement creates countless new possibilities for automation in the logistics sector. These possibilities include order fulfillment and parcel sorting, palletization and depalletization, item picking from shelves, void filling, dimensioning, and counting and inspection.
Order fulfillment and parcel sorting.
Order fulfillment comprises the whole process of receiving, counting, inventorying, picking, packing, sorting, and shipping. Consider robotic picking and sorting.
Imagine a line with moving parcels that need to be sorted by a vision-guided robot according to certain criteria. If the robot is equipped with standard 3D vision — a structured light system, for instance — the conveyor needs to be stopped for each scan acquisition, which leads to longer cycle times and lower throughput. With the parallel structured light technology, the conveyor does not have to stop and the parcels can be scanned one after another in constant motion, without interruption.
This approach has been implemented by Dedoles, a major e-commerce retailer in central Europe that
fulfills more than 100,000 items daily. The automation process starts with parcels being placed on a conveyor belt and moving toward a robot to be picked by it. When each parcel gets within reach of the robot, a parallel
structured light 3D camera scans the parcel and sends its precise 3D coordinates to the robot, which grabs it using a custom-made vacuum gripper. Simultaneously, a 3D scanner scans the QR code placed on the parcel and, based on the information from the code, the robot sorts the parcel and places it into the right container for the delivery company. All of these steps can happen without interruption or human intervention.
Palletization and depalletization. 3D scanning in motion opens up new possibilities for palletization and depalletization. During palletization, which is similar to the parcel scanning, the new technology recognizes boxes moving
on a conveyor belt toward a robot, which, based on the received 3D data, picks them from the conveyor and places them on a pallet for further logistics processing.
Luggage is 3D-scanned for dimensioning purposes. A curved conveyor, irregular movement, and vibrations have no effect on the quality of 3D data. Courtesy of Photoneo.
A contrary process is depalletization, or the robotic unloading of pallets laden with boxes and placing them on a conveyor. Here, the pallets are stationary, but the 3D camera can be attached to the robotic arm and can scan the boxes during the robot’s movement to provide a continuous stream of 3D data without the need to stop for each scan acquisition. This way, the camera can scan the pallet and choose the right box to pick next while the robot is returning to the picking position after placing the previous box on the conveyor.
Item picking from shelves. The hand-eye approach can also be combined with mobile robotics to enable fast and efficient picking of mixed and randomly placed items from shelves. By attaching a parallel structured light camera to the robotic arm of an autonomous mobile robot, each item on the shelf can be approached and scanned from an appropriate distance, angle, and position while the robot moves.
This is a very effective solution for applications that require the picking of various types of items placed on a shelf system and then the placing of them into a container for further processing. The camera mounted to the robotic arm scans the area of the item’s expected location and identifies its exact position. Based on the 3D data, the system localizes the item and navigates the robot to pick it. The technology enables the scan acquisition to happen during the robot’s movement while it is returning from the container back to the shelves.
An example of hand-eye coordination for robotic picking of items from shelves. A 3D camera scans products while the robot moves. Courtesy of Photoneo.
Void filling. The importance of 3D vision rises exponentially in packaging applications, where precise measurements are critical. An important step in preparing goods for shipment is filling the void in the packaging box with cushioning material to prevent movement or damage to the product inside. This process requires determining the box’s volume to calculate the appropriate amount of the filling material, excluding the space occupied by the product.
The 3D scanning and precise measurement of the box’s volume can now be performed in motion — placing the boxes on a moving conveyor, scanning them, and filling them without interruption. This increases throughput and
efficiency while ensuring stable continuity of the process.
Dimensioning. Automated dimensioning involves measuring the width, length, and height of an object or its enclosing box using a vision system. Dimensioning is most often found in package handling, customer billing, pallet dimensioning, or sorting. When boxes are placed on a moving conveyor belt, standard 3D-sensing technologies struggle with providing high-accuracy 3D data at a high speed. Line-scanning systems, for instance, are very fast, but their accuracy and scanning distance are limited. The challenge of capturing fast movement may even be intensified by vibrations or irregular motion caused by curved conveyors.
As their name suggests, line scanners or profilometers capture one line at a time. When the movement is linear, synchronization with the conveyor belt allows composing a full point cloud of the object by placing the line segments next to each other.
Vibrations can cause problems, especially when the object is not perfectly rigid. Due to vibrations, the line segments may be placed at different heights or depths, which results in incorrect data. In contrast to this, area-snapshot 3D cameras capture the whole object at a time; therefore, the vibration does not compromise the final data.
The traditional approach to achieve satisfactory results would be to deploy multiple synchronized vision systems. However, by using two synchronized parallel structured light cameras, a
270º model of each item can be achieved. Based on the 3D data, the system will calculate precise dimensions of the objects and this information can further be used for the calculation of service costs or optimization of the objects’ placement in a transportation container.
Counting and inspection. Keeping a record of all incoming, stored, and outgoing goods is essential for the smooth running of a logistics facility. Automation of the counting process is key for maximizing efficiency, reliability, and throughput. 3D scanning in motion allows logistics customers to count goods moving on a carrier at a fast speed and without interruption.
Similarly, 3D scanning in motion in high resolution and accuracy opens
up new possibilities for inspection processes. The quality control of goods and boxes can occur in constant
motion as the items are passing through the camera system mounted above the conveyor. The large scanning volume of parallel structured light cameras and their ability to create a “mesh” of point clouds make it possible to create precise and accurate images of oversized objects (1.5 × 1.5 × 1.5 m).
Enabling 3D scanning in motion
The missing piece of the 3D-sensing puzzle is a special CMOS image sensor that features a multi-tap shutter with a mosaic pixel structure. This design
enables the construction of multiple virtual images within one exposure window, allowing the capture of moving scenes without motion blur.
Going into more depth, the sensor is divided into superpixel blocks, which further consist of subpixels. Each pixel, or a group of pixels in this block architecture, can be modulated by a different signal within a single exposure.
This is a completely different
approach than that of TOF systems, where all pixels are modulated in the same way and not individually. Though TOF systems also use a multi-tap method with different shutter timings in one exposure window, which enables fast scanning speed, their pixel modulation method results in low resolution.
The parallel structured light approach creates hybrid structured light patterns on the sensor side by combining two types of scanning methods — binary code scanning and phase code scanning. This rare combination results in the lateral resolution reaching up to
2 MP and a depth accuracy of up to 150 to 900 μm across different scanning ranges.
Because the parallel structured light technology uses laser projection, it can apply a very narrow bandpass optical
filter to the sensor to pass only the laser’s narrow spectral range and block the remaining ambient light. At the same time, the 3D reconstruction is
not affected by the laser speckle phenomenon — a typical “noise”-like phenomenon that degrades an image when laser light is reflected from a
surface with a complex microstructure.
With these advancements in CMOS technology and 3D scanning, it is now possible to capture objects moving up to 144 kph (89 mph) in a continuous stream of high-quality 3D data — even in applications involving vibrations or random object motion.
Meet the authors
Andrea Pufflerova is public relations specialist at Photoneo. She writes technological articles on smart automation that is powered by robotic vision and intelligence. Pufflerova has a master’s degree from the University of Vienna; email: [email protected].
Marcel Švec is vice president of
strategy and products at Photoneo. He is responsible for the company’s product portfolio, including 3D sensors and smart automation solutions. He received a master’s degree in artificial intelligence from the Comenius University in Bratislava; email: [email protected].