Europe Develops Multimodal Hyperspectral Quality Control Systems

IVAN NIKITSKIY, EUROPEAN PHOTONICS INDUSTRY CONSORTIUM (EPIC)

Despite rising geopolitical and economic tensions, the global economy is still an intertwined patchwork of competing and complementary regional markets, each with its own unique strengths and challenges. The strengths and successes of these regional economies depend, in part, on their ability to maintain a unified vision and unified goals regarding how they source materials, manufacture products, and innovate technologies.

It was with this unified vision in mind that a consortium of 16 European organizations launched the MULTIPLE project in December 2019 with €5.7 million ($5.9 million) in European Union (EU) funding under the European Commission’s Horizon 2020 program.

The MULTIPLE project’s collaborative model aims to strengthen the photonics manufacturing base in the European Union by developing and implementing cutting-edge snapshot hyperspectral filters (top), laser-based IR analyzers (bottom left), and organicelectronics- based sensors (bottom right).

Short for “Multimodal Spectral Sensors and Orchestrated Deep Models for Integrated Process Optimization,” MULTIPLE’s high-level aim is to increase the competitiveness, flexibility, and efficiency of Europe’s photonics manufacturing base while enhancing the quality of its products. Its manufacturing solutions will allow Europe’s machine vision and image processing equipment suppliers to better compete in global markets.

The project aims to develop cost-effective, multimodal monitoring solutions based on cutting-edge organic-electronics-based sensors, snapshot hyperspectral cameras, and dual-aperture imaging to deliver cost-effective spectrometers and camera cores spanning a broad spectral range from the visible through SWIR bands. The project will complement these solutions by using cost-effective laser-based chemometric sensors operating in the mid-IR.

The MULTIPLE project further aims to augment these hardware modules with cloud-, big data-, and deep learning-based applications to allow agile development and orchestration of complex AI-based models.

MULTIPLE helped to improve quality control for a woodworking operation by measuring parameters such as humidity, density, and the elasticity of the raw input material, and by searching for defects. Courtesy of MULTIPLE.

MULTIPLE implemented sensors to improve the efficiency of steel production by optimizing furnace combustion, precisely controlling the furnace temperature, and monitoring the temperature of the slab in the rolling mill. Courtesy of MULTIPLE.

The conching process involves mixing, stirring, and agitating a chocolate mixture for extended periods to improve the product’s flavor and texture. Imagers from two MULTIPLE collaborators helped to improve quality control during this process. Courtesy of MULTIPLE.

Addressing challenges on this scale would be too expensive for individual companies or organizations. The idea underlying the MULTIPLE project is to identify a demand for solutions from select applications and address them through the implementation of cameras and sensors produced by multiple providers. The project further aims to develop common software solutions to support these multimodal systems and combine them with AI analysis and cloud data storage to enable remote access and control by any end user.

MULTIPLE partners and products

With its launch, MULTIPLE brought together the European technology centers Aimen and imec; technology developers Airoptic, Senorics, SCM, IRIS, Photonfocus, Tematys, Abraia, Monom, and Russula; and technology end users Royo, Jotis, and Laminoires des Landes.

From a technical perspective, the project is based on three main pillars: the development of advanced spectral sensors for volume production, the deployment of a scalable edge cloud infrastructure, and the provision of services for process modeling and optimization based on key performance indicators and dashboards.

The first kind of sensor developed by MULTIPLE enabled snapshot hyper-spectral cameras that covered the visible and SWIR ranges. A hyperspectral camera differs from a standard real-time viewing camera in the way it samples the spectrum. Instead of three channels, hundreds of spectral narrow bands are available in a hyperspectral sensor, and each pixel carries its own spectral signature. The novelty of the solutions enabled by the project is that they use spectrally resolved detectors that allow a hyperspectral camera to have the same form factor as an industrial RGB camera.

Traditionally, hyperspectral cameras have relied on line-scan technology, in which only single lines of light progressively enter the camera via expressive elements used to split the light into various wavelengths. However, special scanning methods are required to create a hyperspectral camera’s datacube. Specifically, as image data goes to the detector, a tunable filter or filter wheel allows only a single wave at a time to pass through.

The MULTIPLE project is now developing a hyperspectral camera in which the filters are placed on top of the pixels, obviating the need for more complex optical arrangements.

Since its launch in 2019, the project has already successfully implemented these multimodal production-monitoring solutions in steel production, woodworking, and industrial food production.

Soon to become commercial, many MULTIPLE technologies are now looking to expand into new use cases.

MULTIPLE use cases

Steel manufacturing operations aim to improve efficiency by optimizing furnace combustion, precisely controlling the furnace temperature, and monitoring the temperature of the slab in the rolling mill. Heating the furnace accounts for 80% of energy consumption in steel plants. Therefore, it is vitally important to optimize energy efficiency by creating a digital model of how the furnace is performing in real time.

To this end, a dual-band visible-SWIR hyperspectral camera developed by MULTIPLE was installed in the rolling stand of one steel mill. Information detected in the visible range can help to monitor the quality of the steel alloy, while the SWIR camera can estimate the temperature of the furnace (to optimize energy efficiency) and the steel slab (to gauge quality). An additional monitoring solution leveraging three laser-based MULTIPLE spectrometers provides highly sensitive and selective gas analysis to reduce CO2 emissions. The information gathered from the camera and spectrometers is collected and stored, allowing the application of various AI models at different stages to improve the overall efficiency of the plant.

When combined with deep learning methods, advanced machine vision, and cloud infrastructure, these sensor devices will enable European industry to meet the need for integrated industrial monitoring and control.

Woodworking processes have provided MULTIPLE with the opportunity to develop technol- ogies that measure parameters such as humidity, density, and the elasticity of the raw input material — both before and during processing. The project’s solutions will also monitor for defect detection and color inspection of the finished product. Accordingly, Aimen has performed preliminary tests using imec’s SNAPSCAN VNIR hyperspectral camera to evaluate Royo’s wooden board samples to learn whether defects can be detected using this type of camera. Although promising results have been attained, more data is needed to decrease the model error and obtain a more representative model.

Chocolate production has benefited from MULTIPLE’s focus on improving quality control of the conching process, which involves mixing, stirring, and agitating the chocolate mixture for a long period of time to improve flavor and texture. As part of this collaborative work, IRIS, a Berlin-based manufacturer of infrared and intelligent sensors, installed two sensors at Jotis, one of the largest food producers in Greece, to benchmark the quality of chocolate products. One sensor offers a spectral range encompassing 256 wavelengths between 900 and 1700 nm, and a second sensor, developed by Senorics, has a spectral range of 1200 to 1700 nm encompassing 16 wavelengths.

While hyperspectral imaging is a powerful tool for process analysis, its high cost hinders its adoption. One way to reduce cost is to reduce system complexity, which Senorics has achieved by converting its hyperspectral imaging device to a point sensor and reducing the number of its spectral bands to 16, a number that is sufficient for analyzing the processing parameters of liquid chocolate during production. By reducing complexity, the camera’s chip design provides less data, which simplifies data analysis.

Despite the trimmed-down design, each spectral band of the SWIR camera provides different information that can be translated into relevant process parameters such as composition, texture, thermal gradient, temperature, colorimetry, and shape. However, estimating measurements from this raw spectral data can be challenging because it requires deep learning methods and advanced machine vision. This, in turn, requires a cloud infrastructure to manage the high volume of data that is required to train the models.

MULTIPLE is also developing various cloud services to support such applications, such as AI for developing and maintaining deep learning models for entire workflows, ranging from data storage to model development and deployment, to production services that support trends, key performance indicators, and dashboards.

MULTIPLE outlook

MULTIPLE’s collaborative model continues to innovate ways to strengthen the photonics manufacturing base in the EU by implementing cutting-edge organic-electronics-based sensors, snapshot hyperspectral cameras, and dual-aperture imaging to increase the cost-efficiency, flexibility, and productivity of manufacturing in Europe. When combined with deep learning methods, advanced machine vision, and cloud infrastructure, these sensor devices — which cover a wide spectral range from the visible down to the SWIR band — will enable European industry to meet the need for integrated industrial monitoring and control.

The main building blocks of MULTIPLE’s cloud infrastructure are smart edge devices that are capable of deploying deep learning models and that can be connected to any spectral sensor following standards such as USB3, GAI, or Bluetooth. The devices can do the processing and publish the results, together with the camera and processing algorithm settings, via an OPC UA (unified architecture) server. This process enables communication between devices and also with machines on the shop floor, as well as connectivity to the cloud. HSI: hyperspectral imaging. Courtesy of of MULTIPLE.

The trials of the project’s systems have helped to improve quality control in steel production, woodworking, and select food production use cases. With the promising results obtained to date, there is every reason to believe that European manufacturers will soon be able to implement such multimodal hyperspectral control systems in other fast-growing applications, such as environmental monitoring, predictive maintenance, waste management, smart farming, forensic science, surveillance, and the production of textiles, pharmaceuticals, and hydrocarbons.