If a traditional picture is worth a thousand words, what’s the value of a 3D image? Quite a bit when it comes to automation. Consider a robot picking a part out of a bin in an automated assembly line.
“If you want to accurately grab a pinion out of a rack, 3D vision can be used for robotic guidance,” said Bob Tremblay, senior manager of product marketing for the 3D business unit of Cognex Corp., a machine vision system maker in Natick, Mass. “You can first validate it’s the correct part. Then you can validate the location of the part: left, right, up, down, and if it’s tilted in any of the three axes.”
Recognition of complex parts of various sizes is possible with 3D vision. Courtesy of ISRA Vision.
Getting part type, location, and orientation correct is critical to inspection, handling, and other automated tasks. The additional data in a 3D image also makes it possible to execute challenging tasks, such as optical character recognition of the lettering on tires. What’s more, 3D vision makes it easier to distinguish between robots or other machines and people — an important capability if robots and people are to mingle freely.
For automation, 3D vision needs to be fast, robust, and precise, but not costly. In the future, the vision systems will need to detect smaller and more complex objects, which, in turn, will require greater resolution.
According to the 2017 report “3D Machine Vision Market — Forecasts from 2017 to 2022” from Knowledge Sourcing Intelligence, the 3D machine vision market should nearly double, from $1.37 billion in 2017 to $2.43 billion in 2022. Increasing applications in the automotive and electronics industries are the main
drivers of growth, with technology advances in cameras and imaging the main contributors, according to the
report.
There are multiple 3D vision implementations, each with its own strengths and weaknesses. Cognex, for instance, uses an approach based on a laser profiler, a technique that projects a laser beam onto a surface. That provides a 2D snapshot of the object, and stitching these together yields a 3D image. The technique can be used to determine if a shaft is centered in an opening, if a bead of adhesive has been laid down properly, or to perform other in-process quality control checks in manufacturing.
Among the challenging automation tasks 3D vision can perform is reliably reading the black-on-black
letters identifying a tire. Courtesy of Cognex.
The Cognex 3D profiler allows an inspection to be completed on-site in seconds. In contrast, a touch system to mechanically profile an object might take 30 minutes and could require moving a part to another building. “You can inspect all of your parts and verify that they’re within spec,” Tremblay said. “The systems are capable of giving you micron-level precision, depending on the field of view.”
Steroscopic approaches
A bin-picking product offered by ISRA Vision AG of Darmstadt, Germany, is an example of another approach to 3D vision. The company uses a stereoscopic approach, with two cameras and up to seven laser lines, according to Tolga Sarraf, regional sales manager for ISRA’s 3D machine vision. Each camera yields a slightly different image. This information, along with knowing where the cameras are in relation to each other, produces 3D data.
A two-camera 3D imaging system captures the volume of a container. Courtesy of ISRA Vision.
This approach was chosen, in part, because in 3D vision, as is the case in 2D techniques, lighting is critical to success. Parts may be dark, shiny, rusty, or otherwise variable in appearance, but the bin-picking process must be reliable. Going with a laser eliminates this source of variation and makes the 3D vision process more consistent. “We are independent and don’t need external illumination, because our laser is the illumination,” Sarraf said.
Most machine vision applications are 2D, but Sarraf predicts that in the future many would move to 3D. The hardware and software for the 3D approach has dropped in price and improved in capability. In addition, 3D techniques yield much more information, which makes it possible to perform tasks such as picking unsorted parts out of a bin. However, 3D vision does tend to be both slower and more expensive than the 2D technology it replaces.
Sarraf said the speed should improve over time because of the arrival of more powerful processors and the continued improvement of algorithms used to extract features and dimensions.
Less expensive hardware should exert downward pressure on the cost. But, the cost question can be complicated. Easier-to-use and more powerful software, for instance, can be more expensive. It also can allow untrained personnel to set up and run the inspection process, leading to a lower overall solution cost. In the case of bin picking, being able to correctly select specific parts out of a random pile means the parts do not have to be sorted first. Cutting out an expensive sorting operation may mean that the total cost of the operation could be lower, even if the vision solution costs more.
With 3D imaging, a system can recognize multiple parts to grip and pick, minimizing cycle times. Courtesy of ISRA Vision.
Sarraf noted that this may be a case where a robotic vision system replaces human labor, which has hidden costs of its own. A person assigned to taking a part out of a box and putting it on a conveyor or otherwise getting it into an automated system may repeat that motion again and again. If the weight of each object gets up to 10 to
20 lbs, both the weight and repetitive nature of the motion can lead to mistakes, not to mention aches, pains, and injuries.
“This is not very healthy,” Sarraf said, “therefore, it is also one reason why companies should do bin picking in order to help minimize such work.”
Time-of-flight systems
A third and final set of approaches to 3D vision comes from Teledyne e2v Ltd. of Chelmsford, England. The company makes 3D laser triangulation and time-of-flight products. With a very fast sensor, 3D laser triangulation offers high accuracy but only works at short distances. Time-of-flight products are not as accurate, but can work at longer distances, according to Ha La Do Thu, marketing manager for Teledyne e2v’s professional imaging group.
A 3D depth map image acquired with a 1.3-megapixel time-of-flight demonstrator. Information about the distance from the camera to various points in the image can be used to distinguish between people and objects. Courtesy of Teledyne e2v Professional Imaging.
The ability to have the sensor some distance from the object of interest is important in some applications. “We have some 3D in the outdoor environment and [time-of-flight] enables robustness to the ambient light,” Do Thu said.
As the name implies, the time-of-flight approach measures how long it takes for a light signal to travel out to and back from an object on a pixel-by-pixel basis. From this, the distance is calculated, and 3D information
derived.
According to Do Thu, Teledyne e2v sees time-of-flight 3D vision as being increasingly important. The ability to work at a distance is necessary when it comes to complex interactions on the factory floor, particularly for situations where robots and people are free to move about. In that case, determining what is a robot and what is a human must be done so there is enough space to avoid a collision and bodily injury.
Because they capture information from three dimensions instead of just two, 3D vision systems generate more data. That need not be much of an extra processing burden if the additional data comes from a few points or the camera only outputs distance data, according to Pierre Fereyre, technical manager for image sensor design at Teledyne e2v.
However, if a vision system produces a 3D point cloud, then the computational, networking, and storage burden may be much larger than is the case with 2D inspection. This load is likely to grow in the future, in part because of increasing 3D resolution.
Teledyne e2v, for instance, is working on a high-resolution platform of more than 640 × 480 pixels, or about a megapixel, in its time-of-flight products. Eventually, 3D sensors of a megapixel or more should become available.
To do that, the pixel size for time-of-flight sensors will need to shrink. This will allow more compact and less expensive optics to be used, but it may require changes in the structure of the pixel. Currently, a state-of-the-art 2D sensor has a pixel size of a few microns. A time-of-flight sensor pixel measures tens of microns, in part because extra processing must be done in each pixel to produce 3D information.
“We are a step back compared to the 2D in terms of pixel pitch because to do 3D time-of-flight pixels we need more transistors,” Fereyre said.
Do Thu predicts that 3D vision will become increasingly common. However, don’t look for the current
mix of approaches to settle down into a single dominant one.
“There will not be one technique that will win over others,” Do Thu said. “We will still have laser triangulation, we will still have stereovision and time-of-flight. It just depends on the application.”