Dr. Lutz Kreutzer, MVTEC Software GmbH
Imaging and machine vision are multipurpose technologies widely used in industry, science, medicine and surveillance. More and more, vision applications are requiring improved production processes and the capacity to address quality issues.
Faster algorithms and better use of current hardware, such as multicore computers, are key for imaging software. Also needed are new matching technologies designed to robustly and reliably find objects or workpieces, even in images with strong perspective distortions.
Moreover, 3-D vision methods and processing of extremely large images are increasing in importance. Advanced methods allow for identification, such as reading of the bar and data codes increasingly used in speed and tolerance.
An example of machine vision software that fulfills all these requirements is Halcon 9.0 by MVTec Software GmbH.
Image preprocessing: GPU or frame grabber?
There has been much discussion about the possibility of outsourcing image preprocessing to the computer’s GPU. But for practical use, this does not make sense. Indeed, on GPUs, processing steps can occur at high speeds, but no currently available hardware enables sufficiently fast data transfer. Thus, every application is decelerated by the twice-slow transfer times, so GPU image preprocessing should not be the first option.
But on a frame grabber, free computing capacity can be used without additional data transfer, so the software must enable image preprocessing in real time. The software also must focus primarily on increasing the speed of the algorithms.
Halcon 9.0 provides a Visual Applets interface that enables image preprocessing in real time with frame grabbers by Silicon Software.
With the rollout of the first multicore machines, MVTec developers began to think about an automatic operator parallelization (AOP) to process images. Halcon was well prepared when, a few years ago, the first multicore processors came on the market. The well-established AOP was part of this technology.
Now a standard
Multicore processors now have become standard for PC hardware, and modern machine vision software makes optimal use of them. With version 9.0, AOP again sped up by 20 percent. The software automatically finds the number of available cores. The image then is split into subimages, which are automatically delivered to a corresponding number of threads. After processing, the data are merged to achieve the complete results. As the number of processors increases, so does the speed (Figure 1).
Figure 1. When executing an operator on a quad-core computer, Halcon automatically splits the image into four parts, which are then processed in parallel by four threads executing the operator. *1/1.96/2.90/3.79/4.51/5.48/6.34/6.93 of the operator median – image with mask size 13 × 13 on a 2 Quad-Core Intel Xeon E5345, 2.33 Ghz, image size 1280 × 1024. Note that the reachable speedup generally depends on the used Halcon operator and the image size. Images courtesy of MVTec Software GmbH.
To meet these requirements, it is necessary to parallelize not only filters but also operators and methods important for a vast number of industrial applications. This includes methods for matching, 3-D matching, subpixel extraction and fast Fourier transform.
Another advantage is the ability to pre-select a region of interest as a free form in an image of any orientation. If the software can process only a preselected area of the image, there is a dramatic reduction in processing time. This benefit increases if the software can process arrays of images in parallel as well as arrays of regions caused by segmentation – e.g., optical character recognition or blob analysis – and arrays of subpixel-accurate outlines, as Halcon does.
However, parallelization makes sense only if enough memory throughput is available; thus, the performance depends on the hardware. Halcon automatically identifies the hardware environment, deciding which algorithms will parallelize and which will not, and avoiding unnecessary overhead.
Multichannel images such as color images are becoming increasingly important, and software can parallelize them with an unlimited number of channels.
Comprehensive 3-D vision
In the field of robotics, machine vision is growing more and more important – particularly comprehensive 3-D vision, an umbrella term for a collection of technologies used to determine the orientation, or pose, of a given object or to reconstruct its shape. Halcon provides the following technologies for 3-D vision:
With 3-D camera calibration, the relationship between camera, object and, if required, robot is established. The calibration should work for area- as well as for line-scan cameras. So-called internal and external camera parameters map the image coordinates to world coordinates, and robot control becomes easier. Calibration is essential for highly accurate and flexible measurements.
3-D matching enables recognizing and determining the pose of arbitrary objects with a single camera. The object is represented by its computer-aided design model. Multiple 2-D views matched to the object’s image can detect its position and orientation using Halcon’s shape-based matching technology extended to 3-D.
Figure 2. Halcon’s descriptor-based matching detects perspective-distorted areas. Interest points are detected where gray values are clearly differentiated from neighboring areas, such as brightness, curvature, corners and spots.
It was only a matter of time until industry requested perspective matching – matching perspective-distorted objects with only one camera. This technique can be based on the detection of interest points where gray values are clearly differentiated from neighboring areas – brightness, curvature, corners, spots. Planar objects such as prints with texture can be located quickly in any pose and tilt (Figure 2). Another method of perspective matching recognizes perspective-distorted objects by shape-based technology. Workpieces and objects with distinct edge- and area-accented elements – such as a car door – can be identified by this method with accuracy, reliability and robustness (Figure 3).
Figure 3. Halcon’s perspective-deformable matching recognizes perspective-distorted objects – such as a car door – with distinct edge- and area-accented elements.
An object within a perspective-distorted image can be more quickly and easily determined with a single camera if the object has significant circles or rectangles; e.g., using the known size of the circle or rectangle to calculate the object’s distance and tilt angle with respect to the calibrated camera.
With binocular stereo, the 3-D coordinates of visible points on an object’s surface can be determined by calculating the disparity of different points of view based on a two-camera setup. During stereo processing, the problem often is that complete areas do not have any texture. To close this information gap, MVTec developed the multigrid stereo, which eliminates the disadvantages of the conventional stereo method. After processing by multigrid stereo, the areas without information (Figure 4, left, dark areas) appear as proper edges and structures (right). Thus, multigrid stereo can bridge texture gaps in stereo images and deliver highly accurate results.
Figure 4. With multigrid stereo, the disadvantages of conventional correlation-based stereo methods are eliminated. After multigrid stereo processing, the areas without information appear as proper edges and structures.
The depth-from-focus method is especially suitable for very small objects. With the height adjustment of the camera, the 3-D conditions of the object can be determined by extracting distance information and calculating the focus of all pixels in an image. Photometric stereo acquires multiple images with illumination from different orientations. In this case, the depth information is reconstructed by using the reflectance features of the object.
In the case of objects without any texture, the sheet-of-light method is suitable. This means measuring an elevation profile of the object by reconstructing the projected line of laser light on the object, generating a 2 1/2-D model.
Bar and data codes
Bar-code reading has been ubiquitous for a long time, and data-code reading is a growing technology worldwide. Ideally, a data code consists of a dot print area composing the actual code, and a frame for orientation and pose identification of the code, called the finder pattern. In practice, important parts can be overprinted, not printed, defocused or damaged by transport and other mechanical influences. For most data-code readers, such a defective code often is not readable. But Halcon enables the reading of such damaged codes, even if the whole finder pattern is missing. The software reads ECC200, QR and PDF417 of each size with elements even smaller than 2 × 2 pixels.
It is important for bar codes to be read in any orientation. Today’s newest software can read a bar distance of only 1.5 pixels.
Processing large images
Software that can process images of more than 32 × 32 k – long anticipated by the industry – is finally here. The size of images is not limited, which is especially interesting for the high-resolution line-scan cameras that are deployed in print and electronic industries for print and component inspection. If this technology is combined with fast parallel processing, real-time processing will be achieved without outsourcing image preprocessing, even with a large amount of data. Thus, programming is significantly easier and faster, and the application can run trouble-free.
IDE with high usability
Machine vision software packages must be comprehensive not only in technology but also in usability. Moreover, they must provide an integrated development environment (IDE) to significantly speed up programming. Halcon’s IDE allows the user to quickly implement an application, improving time to market.
Beyond rapid application development, a machine vision IDE should directly export the application code to C++, C, C# or Visual Basic. Thus, a machine vision application can be directly included in a control program. The IDE also enables the running and integration of external procedures created by the programmer as separate files or programmed by different developers. External procedures are supported by password protection.
Halcon’s IDE runs under Windows, Linux and Unix and provides a development interface, a text editor, a compiler or interpreter, a linker, a debugger and a source formatter.
Embedded software for sensors, smart cameras
In recent years, vision sensors and smart cameras have conquered the market. For noncomplex applications, these devices are a good alternative to a PC-camera setup. Halcon’s architecture is flexible enough to run on special platforms and portable to various microprocessors/digital signal processors, operating systems and compilers.
Figure 5. Halcon’s embedded software library allows development of software on a PC and, afterward, uploading of the application to the embedded system.
Halcon has an embedded software library to provide the full and comprehensive power of machine vision on embedded systems. It allows developing the software part of a machine vision application on a standard platform, greatly easing the programming of an embedded system (Figure 5).
Meet the author
Dr. Lutz Kreutzer is the manager of PR and marketing at MVTec Software GmbH in Munich, Germany.