As the manufacturing world considers the potential for Industry 4.0, machine vision for industrial robotic guidance continues to be a critical enabling technology for realizing the diverse needs of flexible automation and smart manufacturing, especially now when the technologies and tools for VGR are coming into their full potential. The marriage of machine vision and industrial robotics over the past few decades has become a mainstay in practical smart manufacturing. Cyberphysical systems envisioned as part of the next industrial revolution already exist in the form of VGR applications. However, regardless of the specific vision one may champion as the future of industrial automation, pure technological capability always determines the realities of the plant floor. For machine vision and robotics, the key to successful implementation today and in the near future lies in understanding the scope of the applications already widely realized, the challenges to be addressed, and the technologies available for new and innovative applications. Collaborative robots (cobots) follow strict safety specifications that allow them to operate in close proximity to human workers during cooperative tasks. Courtesy of FANUC CORP. A different kind of robot The industrial robot arm is the prolific device that is most familiar in a factory-floor setting. Flexibility is inherent in that the program can be changed on demand and even dynamically controlled or modified in real time based on external input. A recent development in industrial robot technology is the collaborative robot or “cobot.” Cobot arms are basically identical in basic function to their traditional robot cousins, but they have additional safety features that reduce the potential for injury when operating close to a human without guards or fencing. This added safety is accomplished primarily by incorporating sensing in the robot to detect external contact that exceeds a prescribed limit, and by limiting the speed of motion so as not to impart a contact force stronger than the required limit. Safety standards refer to this as “power and force limiting,” although other approved methods such as proximity sensing also exist that allow robots to work safely without fences. Manipulating random mixed product in bins or totes, or on a conveyor, is a challenging vision-guided robotics (VGR) logistics application. Courtesy of FANUC CORP. Machine vision consists of components and software programs that execute image acquisition and analysis to perform a wide range of industrial tasks, including inspection, measurement, and robotic guidance. Machine vision is of course the technology that allows robots to “see.” To be clear, however, the VGR systems widely available today identify the location of discrete, known (or at best, narrowly generalized) objects or features, and do not see in the context of recognizing diverse scenes as a human can. Industrial machine vision for VGR borrows from security imaging, autonomous vehicle guidance, and other similar applications. Machine vision has become a prolific enabling technology for industrial robot arms across a broad rand of applications. Courtesy of FANUC CORP. Successful VGR applications span virtually all industries and processes. Consider this enabling technology in any circumstance in which there would be a benefit to automating the manipulation or processing of individual objects or products (without the need for consistent presentation of the part), with the ability to quickly change parts or products. Assembly, packaging, and palletizing applications — as well as welding, sealing, and machine tending applications — are prominent in the realm of machine vision robotic guidance. These capabilities are in demand in many areas, including the food processing and packaging industries, and pharmaceutical and medical device manufacturing. Machine vision for guidance can be classified in one sense in terms of the degrees of freedom in the pose of the position points provided to the robot — 2D guidance that provides a location in a single plane, or 3D guidance that locates points in space within all degrees of motion. 2D guidance The most common implementation of machine vision for guidance is the 2D location of parts or objects presented in a single plane. With proper imaging and lighting, most standard machine vision products can identify individual objects in a field of view and provide an XY position with rotation to which a robot can be programmatically moved to pick up or place a part. Even when the product is in motion — on a conveyor, for example — the complexity of the application can still be relatively low and the potential for success very high. With 2D, single-plane calibration for VGR, error in position reporting will be introduced when an object is presented at a different height than the original calibration plane. Courtesy of FANUC CORP. In executing 2D guidance, the machine vision task is like other industrial vision applications. The system is configured to isolate an individual part or product in the image, perhaps differentiate or verify that the product is correct and in a desired orientation, and then return the spatial location and rotation of the product relative to a given surface. But where 2D VGR applications often differ from simple inspection tasks at a basic level is in calibration. Field-of-view calibration is a fundamental procedure for most vision components. It involves the mapping of the camera image to a real-world coordinate system. The technique for vision system calibration almost always involves the use of a calibration grid, target, or article that is presented to the camera in the same plane as the object to be located. Basic calibration for a 2D image is usually limited in scope to the exact plane position used during the calibration process. Effectively, that plane must be the same as the surface of the object being imaged. Another consideration in 2D imaging is the effect of parallax when using wide-angle lenses. The resulting image can result in some mislocation of an object caused by stretching of features that occurs at the perimeter of the field of view. In simple 2D guidance, the camera is often calibrated to a world planar coordinate system that is not necessarily the same as the robot’s coordinate system. In this situation, the new object position is communicated as an offset from a taught, nominal position. To use such an offset, the robot tooling position must match the position located by the camera. However, it may be difficult to align the robot tool exactly to the camera location position, and as a result, additional error in pick or place could be introduced. A major improvement is to calibrate the machine vision system to exactly match the robot coordinate frame and incorporate the robot tooling position relative to the location of the part in positional calculations for greater precision. Hand-eye calibration determines the relation between robot and camera coordinates so that camera location results can be easily transformed into robot coordinate points. This representation shows the chain of poses in a typical moving camera system; for example, camHcal is the pose of the calibration plate in camera coordinates. Courtesy of MVTec Co. Ltd.. The calibration of a VGR system to match a robot world coordinate system is sometimes referred to as “hand-eye” calibration, because the “hand” or robot tool is coordinated exactly with the calibration of the “eye,” or the camera. Particularly when a camera is mounted to a robot arm, but also with stationary cameras, this type of calibration and the resulting additional accuracy can improve the performance and ease of use of a VGR system. 3D guidance The biggest benefit of using 3D robotic guidance is its ability to locate an object in any orientation and in any spatial position. This kind of flexibility provides greater opportunities for VGR in a growing base of applications. Objects that are not resting on a specific plane can be reliably located for picking or processing, potentially even when randomly stacked or piled. A broad range of automation applications in assembly, packaging, and machine-tending, for example, can have greater flexibility with 3D guidance, and many industries — including automotive, metal fabrication, logistics, and distribution — profit from the use of VGR technology. The technology is already viable in almost any situation in which an object needs to be located at different heights and/or in different poses or stable resting states within a view. In particular, certain robotic bin-picking applications regularly use 3D imaging to locate randomly organized objects or products to be individually picked and processed. Applications in depalletizing randomly stacked or even mixed packages are expanding. 3D machine vision can successfully support VGR applications that involve randomly oriented objects, and even randomly stacked parts. Courtesy of FANUC CORP. Standard industrial machine vision components that deliver an image with 3D content have become more widely available in recent years. The market growth for these devices is driven by advancements in ease of use, imaging precision, processing speed, and software algorithms that help make the technology accessible and viable for more users over diverse applications. Methods for producing and processing 3D images vary widely by component, and determining which technique is best suited for a given application is one of the tasks when implementing the technology. All 3D images are similar, but the ways in which the basic image is acquired and processed differ. Here are several typical 3D-imaging techniques and brief general characteristics that may influence their selection for a given application: Passive imaging. With a single camera or multicamera, passive 3D imaging uses standard machine vision illumination in the acquisition of the camera’s image and requires no special lighting techniques. It is well suited for 3D VGR, but low-contrast features or lack of texture in the scene can result in less 3D information and poorer resolution. Active imaging. With a single camera or multicamera, active 3D imaging uses patterned lighting to create visible features on the surface of the scene. This increases precision and reliability in the 3D imaging. Systems with active lighting may take a single image of a pattern or multiple images of different patterns, either as an option or as a requirement of the technique. In general, active imaging is the most widely implemented 3D technique in commercial components. Considerations for implementation include speed of acquisition for multi-image systems, precision and robustness of the image production, and potential limitations of the imaging area. Laser triangulation scanning. Also called sheet-of-light scanning, this technology creates a 3D image using a laser line that is imaged by a camera at a set or calibrated angle to the line. The calibrated system calculates the height of points by the position of the line in the field of view and combines many line images into a full 3D representation of the object. Scanning systems are broadly used for 3D measurement and surface analysis, and are also found in robotic guidance applications. Time of flight. Time-of-flight imaging uses specialized imaging sensors and/or camera processing to calculate a distance for each pixel by timing pulsed or phase-modulated light reflections. Imaging speed can be very fast, but the resolution and distance accuracy may be less than other 3D methods. Processing techniques and algorithms for 3D VGR vary as well. The goal of 3D imaging analysis for most common VGR applications is to provide a 3D position to which the robot can move to pick or place an object or perform a process. Some software for 3D VGR is designed to analyze the basic 3D geometry of objects and features in the image to deliver a spatial point and provide a very high degree of flexibility. This type of processing may be more complex to configure or program, and it might be necessary to combine and tune the machine vision tools to achieve best performance on a given set of parts or objects. However, more complex applications — particularly handling mixed, random objects — benefit from this type of processing. Some systems use CAD-based matching to successfully identify and locate an object for 3D VGR (exclusively or in addition to other tools). A 3D representation of the part is prepared in advance, external to the system, and the model is matched to the 3D points in the field of view. The benefits of this location technique are reliability and much greater ease of use; however, it can have limitations. The 3D data available in the scene may not be complete enough to provide a good match to the trained object, and locating mixed objects that can’t be defined in advance can be difficult with this tool. Many industries are seeing high demand for cutting-edge applications in robotic guidance to advance flexible and smart automation. Some of this demand is driven by the need to improve production efficiency and reduce costs. Some is driven by severe shortages in labor resources or by the need to automate tasks that are dangerous. Vision-guided robots and automation are natural technologies to look to in all of these situations. Robotic applications ultimately must involve tooling — for example, mechanical end effectors that grip a part or hold an object or tool. VGR applications (particularly 3D, but 2D as well) have become more successful, in part as a result of advancements in machine vision systems that incorporate more automatic control of grip position. The ability to detect potential interference — either between parts or between the robot tooling and other parts of the automation cell — and make corrections is fundamental to many systems. The ability to manipulate flexible, smart gripping mechanisms is another control capability that may be part of a VGR system. Many industries are seeing high demand for cutting-edge applications in robotic guidance to advance flexible and smart automation. Some of this demand is driven by the need to improve production efficiency and reduce costs. Some is driven by severe shortages in labor resources in certain markets, or by the need to automate tasks that are dangerous. Vision-guided robots and automation are natural technologies to look to in all of these situations. While many applications that need to be addressed are not at all simple, others show promise — if not early success — in certain areas, with future growth potential as technological challenges are met. Picking, sorting, and packaging Logistics, warehousing, and distribution activities involve tasks such as piece picking and case picking in the overall process order fulfillment. Much of the automation in this space centers on human handlers, and there is a need to further automate the process using VGR for picking and sorting. The requirements typically include the ability for a VGR system to identify the location of mixed and unknown products randomly oriented in bins, boxes, or containers. Although limited success has been realized, continued developments and advancements in both 3D VGR tools and robot grip technologies will drive expanded implementation. Deep learning and AI Many applications in machine vision inspection already benefit from new and exciting advancements in deep learning software and systems, but the technology has not yet been broadly incorporated in VGR applications. Basically, it is poorly suited for tasks such as determining location and orientation. However, segments of challenging VGR requirements — such as object identification and differentiation, grip optimization, and path planning — are beginning to be addressed by the technology. In the long term, deep learning likely will find a role in many VGR and machine vision applications. As machine vision applications benefit more and more from alternative imaging techniques, VGR applications are picking up on these technologies and using them to expand capabilities. Advanced color imaging provided by hyperspectral and multispectral cameras is finding its way into VGR for industries such as agriculture and food processing, where robot automation is being successfully enabled by spectral analysis. Applications including robotic harvesting, sorting, recycling, and packaging have seen successful implementations, and more should be on the way. Imaging in the nonvisible IR spectrum is already in use as well. Sorting of waste materials for recycling is one example where NIR and SWIR illumination may be considered. New technologies in 3D imaging — such as light-ray imaging, which produces a multifocus image that can be processed to deliver 3D data — are being introduced for industrial applications and should have potential for VGR. VGR is a mature application area with respect to a broad application base and will remain a mainstay in automation, particularly as the manufacturing marketplace moves toward the efficient, flexible, and data-driven era of Industry 4.0. Future automation designs will require increasingly competent VGR implementation. Meet the author David Dechow is a globally recognized expert in machine vision integration and technologies. He is the principal vision systems architect for Integro Technologies Corp., where he works with sales and engineering teams in the application evaluation and design of complex turnkey machine vision solutions. Dechow is a member of the Automated Imaging Association (AIA) board of directors and is a recipient of the AIA Automated Imaging Achievement Award honoring industry leaders for outstanding career contributions in industrial and/or scientific imaging. He serves on the ASME (American Society of Manufacturing Engineers) Robotic Technologies Advisory Panel; email: ddechow@integro-tech.com. Collaborative robots, machine vision, and safety Industrial robots require physical separation from humans for safe operation. But part of the concept of Industry 5.0 involves the ability for humans to work side by side in close collaboration with robot partners in the production environment — with a collaborative robot or “cobot.” However, a word about practical safety is in order relative to the realities of the technologies available in today’s market. As one considers applications for human-robot collaboration, it is important to understand that cobots in commercial use now, without external safety sensors and control systems, generally do not perceive the presence or location of a human co-worker until physical contact is made. VGR adds to the safety challenge in that a vision-guided robot moves autonomously and less predictably to a greater variety of positions than a “blind” robot, which only moves to fixed preprogrammed positions. A cobot carrying, for example, a welding torch or a cutting blade close to a human may be guided to positions not anticipated by the person and could cause harm before it senses the presence of the human. A complete safety analysis of every application is strongly recommended and is in fact a required part of the safety specifications for collaborative robot implementation.