With the capacity to cover at least 30 contiguous spectral bands between 250 and 2500 nm, hyperspectral cameras can uncover the spectral profiles that uniquely fingerprint materials.
HANK HOGAN, CONTRIBUTING EDITOR
PICVISA Machine Vision
Systems of Spain is striving to make the world cleaner by enabling the automatic sorting of textile waste. This is a critically important task when it comes to meeting regulations aimed
at reducing the contents of landfills
because sorting is a necessary first step in recycling. But standard optical
imaging cannot readily distinguish between textiles.
This is why PICVISA turned to hyperspectral machine vision, capturing the reflection of a material waste stream across many spectral bands in the
infrared region and processing the signal using artificial intelligence (AI). The result is a signature that robots use to discriminate between materials and thus sort the waste stream. Estimates are that only a tiny fraction of textiles are recycled today. The European Union, though, requires all its member countries to implement systems to manage 100% of their textile waste, >16 million tons per year, by Jan. 1, 2025.
Hyperspectral machine vision enables sorting of textiles, a critical necessity for recycling. Courtesy of PICVISA Machine Vision Systems.
“This technology is a game changer. It enables us to identify materials and compositions of garments for preselection before recycling,” said Daniel
Carrero, technical director of PICVISA, in talking about the Specim FX17
camera used in his company’s product. The camera operates in a line-scan mode, collecting hyperspectral data from 900 to 1700 nm.
Canada-based Waste Robotics is applying similar techniques and the same camera to sort plastic waste. High-density polyethylene and polypropylene plastic lids look identical in the visible region but distinct using hyperspectral imaging in the short-wavelength infrared (SWIR).
Falling costs and rising performance in cameras, light sources, and software are all contributing to the increasing use of machine vision hyperspectral cameras for waste sorting. Additional applications seeing a rise in the use of hyperspectral machine vision include sorting food and assessing its quality, spotting contaminants in food such as shell fragments among nuts, determining mineral composition in mining, and monitoring agricultural fields.
A higher level of accuracy
All these tasks have something in common: They require highly accurate identification of materials that look identical to the eye.
“Hyperspectral imaging is applied when all other technologies have been attempted and fail or do not meet the required inspection accuracy,” said Minna Törmälä, Specim global marketing manager.
An RGB image (top) versus a hyperspectral image (bottom) of different plastic types. Courtesy of Specim.
Waste Robotics’ waste sorting robot based on spectral-based real-time material identification. Courtesy of Specim.
A hyperspectral machine vision camera can succeed because it is fundamentally different from a standard RGB camera. The latter gathers information
in three bands — red, green, and blue — in the visible (400- to 700-nm wavelength) region. More complex camera systems may capture additional bands, either in the visible region, the near infrared (NIR), the SWIR, or longer wavelengths. But typically, such multi-
spectral cameras image <10 bands and often only four or five, according to Törmälä. She said that usually the bands are <1000-nm wavelengths, which means the cameras do not acquire the composition information found in the longer-wavelength infrared region.
In contrast, according to the draft IEEE P4001 standard, a hyperspectral camera covers at least 30 contiguous
spectral bands lying between 250 and 2500 nm. The camera may image across hundreds of narrow spectral bands, thereby uncovering a spectral profile that uniquely fingerprints
materials.
Jens Hashagen, business development manager for SWIR imaging for camera maker Allied Vision, said that hyperspectral imaging uses much the same setup as standard imaging, with light sources, filters, lenses, and sensors. The key difference in hyperspectral imaging is a component that sits in front of the sensor and separates the incoming light into its spectral constituents, enabling the detection of what is called a pixel vector that contains more than spatial information.
“Each pixel vector in a hyperspectral image contains a complete spectrum, and together they build a hyperspectral image data cube, allowing for detailed analysis and identification of materials,” Hashagen said.
Another difference between hyperspectral cameras and more traditional ones is the analysis software. The complex nature of the captured spectral signatures can make analysis challenging. The use of AI algorithms improves the reliability of materials classification, making it possible for users with minimal expertise to conduct such analyses at a rate and with an accuracy needed for industrial applications.
The most common spectral separation setup turns a 2D sensor array into a 1D line scanner, with spectral information along the other axes. Thus, the x-axis consists of base spectral band data of a line of points while the y-axis will include the data for different spectral bands of each point. Moving the band across an object yields its spectral information.
Push broom approach
Such a push broom approach, named because the scanning action is like sweeping a broom across the object, produces useful results but creates certain tradeoffs. Albert Tu is a product manager at Hamamatsu, which produces hyperspectral imaging detectors. He said that when performing a line scan, which the push broom method requires, the integration time, or the amount of time spent at each point, may be short to meet throughput requirements.
A white light laser line on a hyperspectral corn sorting machine. Hyperspectral imaging enables precise overall quality assessment and specific grade sorting, considering factors such as corn sweetness through sugar content measurement. Courtesy of NKT Photonics.
Textiles must be recycled according to new regulations enacted by the European Union. Courtesy of PICVISA Machine Vision Systems.
“That means your light is limited,” Tu said. He said that this constraint limits sensitivity.
This example illustrates tradeoffs facing providers who must strike a balance
between cost, resolution/sensitivity,
and throughput time. Improved sensitivity, throughput, or both may, as a
result, require cameras that are too costly for an application.
There are alternatives to the push broom arrangement, with these producing their own set of tradeoffs. Hyperspectral camera maker Photonfocus, for instance, makes devices that capture an area image in a single exposure, called snapshot cameras. Mike Faulkner, sales manager at Photonfocus, said that pixel level spectral separation helps Photonfocus make its cameras smaller and more rugged than the standard push broom approach. Those attributes can be beneficial for machine vision cameras operating in an industrial setting.
“Production lines aren’t exactly the cleanest places, and they tend to not have a lot of space,” Faulkner said.
Photonfocus uses sensors from imec, the research consortium that helps commercialize innovative technologies. According to Wouter Charle, program manager for on-chip spectral imaging, imec makes the sensors by depositing films on each pixel. In this way, each pixel only receives light from a specific optical band. The films are often laid down in blocks, with 4 × 4 and 5 × 5 being common arrangements. Thus, the result is either 16 or 25 spectral channels if the blocks are all the same. However, imec has performed demonstrations with complex mosaics and intricate filter patterns.
A benefit to this approach is that it allows the construction of very compact spectral resolution cameras. Also, using semiconductor fabrication techniques eliminates manually assembled components, which lessens the price and improves reliability.
This in-pixel approach can be applied to different sensor materials, such as CMOS for the visible region, indium
gallium arsenide (InGaAs) for the NIR and SWIR regions, and mercury cadmium telluride for longer wavelengths.
“We turn a regular sensor into a multi- or hyperspectral sensor,” Charle said.
He added that the technology enables true snapshot and spectral imaging. The technique achieves video spectral rate imaging because of its mosaic pattern, such as the 4 × 4 or 5 × 5 blocks mentioned previously. The cost of this approach is spatial sampling resolution in each channel. However, Charle said that algorithms that reverse the mosaic can recover close to full spatial resolution.
As for the detector materials used
in hyperspectral imaging, recent innovations promise better performance. Sony’s SenSWIR technology, introduced in 2021 and commercially available, can cover wavelengths from 400 to 1700 nm. Thus, one camera can provide hyperspectral data in both the visible and SWIR regions. Other companies, such as Emberion, are working on technologies that allow imaging from 400- to 2500-nm wavelengths.
Advancements are also occurring in light sources. Because hyperspectral systems gather data across a spectral band, the best light sources are those that emit illumination across the band. In the past, the technology of choice has been halogen lamps. But they generate heat, which wastes energy, can damage equipment, or harm the product being scanned.
Cold illumination
“We can do ‘cold’ illumination as compared to traditional sources, such as halogen lamps, which produce
a lot of heat,” said Jesper Toft Olsen, product line director for NKT Photonics.
The company makes supercontinuum sources. According to Olsen, these can be for either line or point
illumination and can cover 380- to 2400-nm wavelengths in one light source.
Even though prices have fallen over the years, supercontinuum sources are still more expensive than halogen lamps and LEDs. The latter have seen innovations that have expanded the range of wavelengths available, filling in illumination gaps that hindered machine vision in the SWIR.
Regarding costs in general, industry players report progress. “Systems are coming down in price from hundreds of thousands of dollars to under $10K per unit,” said Barry McDonough, CEO of Hinalea Imaging. He said that today, a hyperspectral camera may cost 5× or so more than a standard RGB machine vision camera.
Looking to the future, PICVISA’s
Carrero said that he would like to see even lower hyperspectral machine
vision prices. Another desire is achieving extended spectral ranges because this would enable more accurate
classification of materials. A third desired advancement is higher spatial resolution and higher speeds. A fourth and last request is for more sensitive cameras because that would reduce the need for illumination. Lighting is the major energy cost overhead of the equipment and it affects equipment durability and performance consistency over time.
“These four points are the most important for the industrialized systems that we supply,” Carrero said.
Progress in these areas would help boost the industrial use of hyperspectral machine vision. Even so, if current systems provide the required spectral information, then the current cost and performance may be worth it for some applications.
For instance, Resonon CEO Rand Swanson said, “We’ve sold more systems for recycling than any other machine vision application by far.”