Machine vision often uses deep learning technologies to identify objects and detect defects. Two new methods make these processes even easier and more transparent.
Mario Bohnacker, MVTec Software GmbH
Technologies based on artificial intelligence (AI) are used in many industries today. For example, deep learning methods based on convolutional neural networks (CNNs) are used in machine vision, making it possible to detect and localize objects and defects in a more targeted manner across the entire industrial value chain. Alternatively, rule-based systems can also be employed. For defect detection, however, these systems may have to cover a large number of error characteristics, which causes a need for an extremely high programming effort.
AI technologies offer a significant advantage. Through targeted training, they learn independently and continuously and are therefore able to reliably recognize new variations of possible defects. The training process is divided into several steps. It is important to first generate and collect a sufficiently large number of training images. The images must show all possible defect types that are to be detected. Afterward, the images are labeled so that they can be clearly assigned to a specific defect class such as “dent” or “scratch.” Only then does the actual training of the underlying CNN follow and use the digital image data.
Classifying defects
Deep learning algorithms can be used in a variety of technologies, depending on the application. During classification, objects or defects are divided into appropriate classes, based on the overall image information that encompasses texture, gray values, or color data. Object detection goes even further. In this case, images are labeled by drawing rectangles around the objects to be identified. This allows multiple
objects of various classes to be identified in one image (for example, evaluating the contents of a shopping basket
in an automated checkout system).
In the process, the deep learning algorithm learns which characteristics can be used to assign an object or defect to a specific class. On this basis, objects or defects can be classified and localized automatically. Semantic segmentation is even more precise. Using this technology, each individual pixel in an image is assigned to a specific class so that entire regions can be divided into classes.
Previous deep-learning-based detection methods share one thing in common: They require a large amount of training data. All images have to be labeled to assign them to a certain error class. When it comes to detecting defects, for example, at least 300 “bad” images are needed for each defect class, depending on the application. This means that the image must show the object in question with the potential defect. Possible defects may include deformations, cracks, scratches, dents, and other conceivable anomalies. One challenge is that for many possible defect types, the necessary number of bad images is often not available because certain types of defects rarely occur in production processes. Add to this the labeling effort required for the images. In the simplest case, the entire image can be assigned to a specific class. However, it may also be necessary to mark the defects in the images, which can require a great deal of time and effort.
Few training images required
An innovative technology called anomaly detection requires significantly fewer training images than other deep learning methods. From 20 to a maximum of 100 images are sufficient to completely train the CNN. Moreover, the process requires no bad images, but only images of the defect-free object. This eliminates the effort of labeling the training data. Another benefit is that training the CNN takes much less time than with other deep-learning-based methods. Just a few minutes are required. The anomaly detection technology localizes defects whose appearance and manifestation were not previously known. This allows inspection tasks to be solved more efficiently and with considerably less effort.
The heat map identifies image regions that are crucial to classification. Courtesy of MVTec Software GmbH.
Deviations from the target state may be based on a wide range of properties, such as the object’s color or structure. For example, a beverage manufacturer can precisely inspect the containers to be filled without having to collect images of all possible defects in advance. Anomaly detection can reliably detect and localize even the tiniest damages to the neck of a bottle that are invisible to the naked eye, such as cracks, fissures, or scratches. During
the process, the software creates an anomaly map. On this map, areas where an anomaly is found are marked with a gray value. By segmenting the map, the position of a defect can be localized down to the last pixel and its size determined.
Along with anomaly detection,
another new feature further simplifies
and optimizes machine vision processes: a heat map based on the gradient-weighted class activation mapping (Grad-CAM) method. This feature solves an age-old problem in the deep learning-based detection and inspection of objects, namely that decision-making largely takes place in the background because CNN is a black box that is difficult to view from the outside. Thus, the classifier’s
decision cannot be tracked directly
during detection. The specific learning
process within the deep learning
network is usually not visible.
Deep learning-based detection is possible using classification, object detection, or semantic segmentation. Courtesy of MVTec Software GmbH.
This lack of visibility can have a serious effect on the quality of the detection results. If an object is classified incorrectly, the cause is difficult to identify and debug, resulting in situations such as in the following practical example: The CNN was trained using various polar and brown bear images, and the deep learning algorithms were initially able to make a reliable distinction between the two types of bears. But when an image of a brown bear against a snowy background was shown, the image was classified as showing a polar bear. How did this error occur?
The new heat map feature reliably detects anomalies. Courtesy of MVTec Software GmbH.
The heat map can solve this problem. This feature, which has been integrated into MVTec HALCON with version 19.11, sheds more light on deep learning processes. It precisely analyzes which image areas significantly influence an object’s assignment to a specific class. A false color representation highlights the relevant regions in the image. This allows conclusions to be drawn about the quality of the training data and makes the CNN’s decision-making process more transparent.
In the bear example, the heat map makes it possible to identify the information that the deep learning network used to differentiate the image. The criterion was obviously not the bear itself but rather the background, meaning the snowy area. This leads to the realization that training was based on false or incomplete data and therefore produced an incorrect result. On this basis, the user can expand the training process and add more images, such as those depicting additional brown bears in various snow and ice scenarios. The training process can now be repeated until the best possible detection rate of the different bears is achieved.
The heat map method is not new, but the solutions available up to now have had a very long execution time. Grad-CAM-based heat map solutions, however, take less than 10 ms to run. This speed makes them ideal for online visualization applications.
In addition, the latest variants now also run on a central processing unit (CPU). This offers a significant advantage for all deep learning users because the need for a performance-hungry graphics processing unit (GPU) is eliminated. As a result, conventional industrial PCs without GPUs can be used to compute the heat map directly on the production line, enabling users to analyze their deep learning network’s class prediction on the fly.
Meet the author
In his capacity as technical product manager at MVTec Software GmbH, Mario Bohnacker is responsible for the technical enhancement of the standard machine vision software HALCON. His duties include identifying technologies that are relevant for future developments, as well as for customers, and turning them into product innovations. Bohnacker also directs the further development of MVTec HALCON; email: [email protected].