Advancements in machine learning and computational power enable identification of similarly colored defects in products.
RAND SWANSON, CASEY SMITH, AND RYAN ANDERSON, RESONON INC.
Hyperspectral imaging has
a long history of successful applications in the field of remote sensing, but only recently has it matured enough to find success in industrial
settings, including applications in pharmaceuticals, food production, manufacturing, and recycling. Although
it was developed decades ago, hyperspectral imaging is not as widely adopted or as represented in literature as is conventional spectroscopy, which has achieved widespread commercial acceptance with an annual market of approximately $10 billion.
A conventional spectrometer separates a beam of light into many narrow spectral bands via a diffractive or refractive element and projects the resultant light onto a row of detectors. Only a narrow range of wavelengths, or spectral band, is incident on any particular detector. Taken together, these spectral bands form a continuous
curve, representing brightness as a function of wavelength. This curve is called the “spectrum” of the light.
A hyperspectral imager can be envisioned as hundreds of spectrometers in a row, with each spectrometer sampling a different spatial region (Figure 1). A two-dimensional hyperspectral image is recorded row by row as the object or imager moves, just as with conventional line-scan imagers. However, with detailed spectral data for all pixels, the capabilities of spectroscopy can be used for every pixel in the image. Recent advancements in machine learning algorithms and computational power have enabled this data to be quickly and accurately analyzed, producing real-time classification and analysis. In machine vision contexts, this means small, similarly colored defects in products can be identified and sorted, variable product can be graded accurately, and manufacturing processes can be tuned based on feedback from the hyperspectral imaging system.
Figure 1. Hyperspectral imaging is spectroscopy for every pixel. Courtesy of Resonon Inc.
The differences in performance between hyperspectral machine vision and conventional imaging are analogous to the difference in performance between using a spectrometer and a light meter. Spectral data enables the ability to distinguish between similar objects, monitor narrow spectral features such as those produced by water or other molecules, and identify color changes imperceptible to the eye — all tasks that cannot be performed nearly as well, if at all, with single-band or even multiband imaging.
Figure 2. A color image showing two types of red candies that are difficult to distinguish. One type is arranged in the shape of a capital ‘I’ not
visible to the naked eye. Machine learning algorithms highlight the differences in spectral data, apparent in Figure 3. Courtesy of Resonon Inc.
These benefits can be appreciated by carefully considering an image of two red candy types arranged in a pattern (Figure 2). Can you pick out the “I” shape? It is difficult to distinguish between the two colors, and conventional-color machine vision would likely do a poor job of discriminating between the types. However, the spectral data for each type is distinct. This difference can be used by machine learning algorithms to identify the types (Figure 3).
Figure 3. Spectral curves for the two types of red candies (top), and an image of the classified candies (bottom). Courtesy of Resonon Inc.
Although this example may be convincing, it is artificial in that the candy types are uniform in shape and color, and there are only two object types. A real-world example that exhibits the advantages of hyperspectral imaging involves sorting foreign material from nuts. Nut sorting is a challenging application for conventional imaging because some foreign materials have colors and shapes similar to nuts.
Additionally, there is considerable natural variability in the color of the nuts as well as in the foreign materials.
In this example of industrial nut sorting and grading, a system consists of a hyperspectral imager, stabilized broadband lighting, and a flip-down reference panel used for periodic calibration. All of these components are mounted above a conveyor belt carrying a single layer of nuts. The hyperspectral imager is interfaced to a data acquisition and analysis computer, along with data from the conveyor belt motion encoder.
Before use, the system is trained. Nuts are arranged across the belt and scanned. Custom software tools differentiate the nuts from the belt, and then the spectral information from the nuts is used as training data. By including nuts of differing shades of color, normal product variations are incorporated into the training. This process is then repeated for the predominant foreign materials and defects, such as hulls, shells, damaged nuts, rocks, and sticks.
Machine learning algorithms use the input training sets to learn the spectral features that best distinguish between good product, poor product, and foreign materials. These prediction models can be easily updated by scanning in ensembles of a new foreign material or good product.
In machine vision contexts, this means small, similarly colored defects in products can be identified and sorted, variable product can be graded accurately, and manufacturing processes can be tuned based on feedback from the hyperspectral imaging system.
Thus, in addition to high-accuracy and reliable sorting, customers are furnished with detailed product and process information that can be used for superior grading and improved process control.
Rules of thumb
Many other applications are possible, but how does one determine whether hyperspectral imaging could be a good solution to a particlur problem? A few guidelines and considerations are discussed below.
Generally speaking, hyperspectral imaging is useful for applications in which the object of interest can be distinguished by its surface properties.
Thus, color matching and finding foreign materials across a product stream make good candidates, whereas locating bones inside meat or finding defects in stacked product are less suitable.
Hyperspectral imaging is a good candidate for when the objects of interest are distinct within a product stream — for example, finding counterfeit products or detecting blemishes in a finished product. Monitoring homogeneous materials, such as well-mixed paint, could be performed with a hyperspectral imager, but a conventional spectrometer would be more cost-effective.
Because hyperspectral imagers record many tens to hundreds of spectral channels per pixel, the technology is slower than conventional line-scan imagers. Thus, speed and resolution must be considered. As an example, some bacteria can be identified with hyperspectral imaging, but the high resolution needed, along with the large throughputs of most food processing systems, makes detection of bacterial
contamination impractical. At the opposite extreme, applications that currently use human vision for identification make strong candidates for hyperspectral imaging.
Additionally, many applications exist outside of the range of human vision. Pharmaceuticals and vitamins are
often similar in color and shape. In
many cases, spectral data at IR wavelengths can be used to accurately distinguish between pills that cannot be differentiated by color (Figure 4). Another example is monitoring moisture content, which affects the quality of the finished product, in food or paper streams.
Figure 4. Pills that cannot be differentiated by color are often easily differentiable using hyperspectral imaging at IR wavelengths (left). Various types of white pills are color-coded (right). Courtesy of Resonon Inc.
When a problem may be solved using hyperspectral imaging, detailed testing is prudent. Academic papers that demonstrate proof of concept are useful, but laboratory success does not always predicate industrial viability. Therefore, experts recommend a three-step process with go/no-go decisions between each step.
First, image and analyze known samples in a laboratory environment. Samples should include good product as well as the defects of primary interest. This step is typically an inexpensive starting point that establishes project feasibility and gives an indication of
the difficulty of the problem.
During this step, operators should expect to work closely with a systems integrator or hyperspectral solutions manufacturer, offering specifics on product, the defects of interest, the goals and priorities of the potential
vision system, and the operating
environment.
Hyperspectral imaging aids in sorting foreign material from nuts.
Assuming feasibility is demonstrated in the first step, the next step typically
involves tests in a production or production-like setting, in which high volumes of product are scanned at production speeds. This provides statistically significant volumes for accurate performance assessment, enables algorithm optimization, and often exposes challenges not appreciated during proof-of-concept testing. At the end of this phase, users will have the performance and cost information needed to make a well-informed go/no-go decision for the third and final step of full installation.
Hyperspectral imaging allows users to identify various materials or colors
for sorting, process control, or grading.
Its heritage is spectroscopy and therefore it is a well-known and powerful tool, scaled massively to be useful for real-world production volumes and speeds. Hyperspectral imaging will generally outperform conventional color or multiband imaging systems and often solves problems too difficult for conventional imaging. Consequently, it offers a wide range of possible applications for improved quality, reduced waste, and enhanced safety in industrial processes.
Meet the authors
Rand Swanson is president of Resonon Inc.; email: [email protected].
Casey Smith is a senior scientist at Resonon, focusing on airborne and machine vision applications.
Ryan Anderson is a software engineer at Resonon and specializes in machine learning.