ABI RAMANAN AND GUSTAV NIPE, IMPACTVISION
Despite being the world’s most globalized industry, agriculture is extremely inefficient. Global supply chains have been mechanized but not information-enabled, and they have not yet fully benefited from the explosion of digital technologies that have helped in other sectors, such as pharmaceuticals or health care.
At the same time, a third of all food produced in the world is wasted. If food waste were a country, it would be the third largest emitter of greenhouse gases in the world, generating about 8 percent of global carbon emissions annually. More than 20 percent of meat produced globally is wasted; for fruits and vegetables, it is a staggering 45 percent.
A hyperspectral benchtop system for fish freshness classification. Courtesy of ImpactVision.
Additionally, food fraud costs the global economy an extra $40 billion per year — and scandals have increased by two-thirds since 2010. This type of crime is growing at such a rapid pace that the U.K.’s Food Standards Agency established the National Food Crime Unit in 2015 specifically to address fraud and adulteration.
The scandal of food waste has gained increased attention over the last few years, culminating when the United Nations General Assembly (UNGA) called for halving per capita global food waste at the retail and consumer levels and reducing food losses along production and supply chains, including postharvest losses, by 2030.
As outlined in Target 12.3 of the UN’s Sustainable Development Goal (SDG) 12, the call to action seeks to “ensure sustainable consumption and production patterns.” A key step to achieving the target for food loss and waste is for governments and companies to set specific reduction targets aligned with SDG 12.3.
A hyperspectral image of a beef steak. The yellow color on it represents fat, and the dark color is lean meat. Other colors are selected randomly to indicate background and other objects in the scene. Courtesy of ImpactVision.
For food producers and retailers, quality control and assurance still rely on manual, lengthy, destructive processes, such as visual inspections, sensory evaluation, and sample-based testing.
Sorting, distribution process improvements
Noninvasive, rapid, objective techniques for monitoring food products are essential to improving the sorting and distribution processes. This is where hyperspectral imaging has great promise. The technique acquires both spectral and spatial data; it then visualizes physical and chemical constituents noninvasively in real time and can facilitate a shift toward supply chains where 100 percent of the products can be tested.
However, the use of hyperspectral imaging for food monitoring must overcome certain challenges, on both the industrial and technical levels, before its full potential can be realized.
Before food companies can adopt a new technology, they have to understand it. Currently, companies are not accustomed to reading plots and statistical values. Information is typically acquired via firmness tests, instruments such as pH meters, or through sensory evaluation such as squeezing, smelling, or looking.
Information from sensors, such as hyperspectral or multispectral cameras, needs to be translated into actionable insights — for example, the number of days of shelf life, freshness categories for fish, or optimal sorting based on a chemical attribute such as the dry matter content (correlated with ripeness) of an avocado. Such insights are valuable for food companies, enabling them to increase yields, reduce waste, and improve sorting for distribution and quality.
Reflectance spectra for the three pixels identified on a beef steak. The green line represents the reflectance of the white paper background, which has the highest reflectance; the blue line represents the white-ish fat of the meat and reflects more than the red line, which represents the lean, dark part of the beef. Courtesy of ImpactVision.
The widespread adoption of this technology will also require a change in procedures. For example, there are several academic papers stating that hyperspectral imaging can be used to classify the tenderness of beef steaks. But in industry today, beef tenderness is mostly graded in the packaging house by beef producers using destructive tests, such as the Warner-Bratzler shear force test.
Consumers, for their part, are not accustomed to purchasing beef based on a tenderness grading, despite valuing that as an important attribute for taste. The business challenge to introducing a superior quality metric in the meat industry requires a shift in procedures by both industry and consumers.
Technical improvements are necessary for the widespread use of the hyperspectral imaging system for food monitoring, such as in illumination. The faster the speed of a conveyor belt, the greater the strength of illumination required for both an area scanner and a line scanner, compared to a single-point spectrometer. Halogen lights, which illuminate over a broad spectrum, have a shorter lifespan, and they generate heat. The traditional solution when more illumination is required is to increase the power, but with halogen lamps this also involves more heat. Too much heat can damage the food item being processed: Sugar melts, for example, and ground pork loses moisture more rapidly.
A hyperspectral image of sugar and foreign objects. Courtesy of ImpactVision.
LED illumination radiates much less heat, making it a better choice. However, current LED technology is insufficient for hyperspectral imaging solutions in the food industry because it does not illuminate light over a broad spectrum. Usually, a LED emits light in a very narrow spectrum, which means a multitude of LEDs would be required to mimic a halogen lamp.
The faster the speed of a conveyor belt, the greater the strength of illumination required for both an area scanner and a line scanner, compared to a single-point spectrometer.
Hyperspectral imaging for food monitoring also requires real-time classification of a hyperspectral cube. Hyperspectral imaging systems have been developed mainly for laboratory use or research environments, with a slow-moving tray. Food companies continuously try to increase the speed of their conveyor belts to optimize throughput, which requires a sorting application where analysis of a hyperspectral image (which can be up to 1 GB in size) needs to happen in less than a second.
An assessment of existing hyperspectral literature finds that application of this technique for fast, production-line speeds is not being addressed by researchers. Service providers such as ImpactVision have to develop custom solutions for clients. This includes dividing the image into subparts so classification of the whole image can happen in parallel.
Data collection presents another challenge. Each food product requires a different calibration. This, in turn, requires product-specific knowledge about how to operate the ground-truth reference measurement, such as a pH meter or shear force test, and how this correlates with spectral data from a hyperspectral image. To develop these algorithms, service providers need to partner with food companies to acquire enough data points for commercially viable solutions.
But there are no large, publicly available spectral data sets of food products with corresponding ground-truth measurements, resulting in a lengthier development process to validate the quality of the classification algorithms that service providers develop.
A final technical obstacle involves foreign object detection. The spatial resolution of hyperspectral imaging could be improved to detect smaller foreign objects. Today, to detect one hair, the camera would need to be less than 10 cm from it, which reduces the field of view of the camera and limits the width of the conveyor belt it can cover. In this instance, more cameras can be added to image a wider area. However, this significantly increases cost.
New solutions are emerging that enable the use of hyperspectral imaging for food monitoring. The technology has existed for more than 20 years but could not be applied in food processing facilities for in-line use until recently. The large volumes of data generated and incumbent classification models made the technique difficult to use for real-time quality control.
The proliferation of easy-to-use neural network machine learning frameworks has made the development of superior nonlinear classification algorithms or models much faster. And combining GPUs that can be programmed with machine learning models allows for rapid calculations that occur in parallel, resulting in faster classification of hyperspectral images.
Classification algorithms
Hyperspectral imaging technology can be used for many varied in-line applications in food processing at distribution centers. ImpactVision has developed software with a variety of uses, including detecting beef pH and color algorithms. The algorithms can reduce in-store meat waste for retailers by accurately calculating the shelf life at the distribution center and optimal form for end use of a product to deliver greater quality and consistency to consumers.
The use of hyperspectral imaging for food monitoring must overcome certain challenges on both the industrial and technical levels before its full potential can be realized.
For example, if beef has a pH of above 5.9 when vacuum packed, it will become DFD (dark, firm, dry) much faster. Muscle with a postrigor pH of 5.9 or greater generally develops some form of dark cutting characteristic. The pH range of normal meat of an unstressed animal is 5.4 to 5.7. DFD meat will have a much higher pH, in the range of 5.9 to 6.5.
The dark-red state of DFD meat in a retail display case has proven to be one of the leading causes of consumer rejection. Consumers see the dark color as indicating meat from an old animal, spoilage, undesirable flavor, toughness, and poor shelf life.
With ImpactVision’s classification algorithms, food retailers can identify high-pH meat earlier in the supply chain. That meat can then be turned into secondary products such as ground beef or mince. In the long term, retailers can identify the source of the issue, such as temperature fluctuations or poor-quality animal feed, and develop tighter product specifications based on objective data.
Beef tenderness classification is another application. Today, there is no imaging, but the system allows a prediction at two days of how tender the meat will be at 14 days, giving another 12 days to optimize end use. Fourteen days is the industry standard time for measuring beef steak tenderness. After 14 days, tenderness values typically do not change.
As the decay process of fresh hake progresses, the skin changes from green to red. With the human eye, it is barely possible to detect the green area. However, a hyperspectral camera can ‘see’ these colors, and the software uses them to classify freshness. Courtesy of ImpactVision.
Prediction data at two days is highly valuable. Decisions about the use of the meat often occur at the two-day mark when beef steaks begin to be shifted for subsequent storage and distribution. This classification can enable dynamic pricing by guaranteeing product tenderness through an objective measurement.
Hyperspectral imaging technology also can detect foreign objects. X-ray machines typically detect objects that are denser than the food substance itself. This creates a challenge for food companies when attempting to detect certain lighter, nonmagnetic materials, such as plastic, bone, paper, and cardboard.
Hyperspectral systems can detect and classify such objects with a high level of accuracy by dividing the image into thousands of pixels and grouping together pixels with similar spectral profiles. This quality-control tool identifies contaminants in real time and can be integrated with a robotic arm or an air-jet blast for instant removal. This can prevent complaints or product recalls further down the supply chain.
Noninvasive classification of avocados is also possible via hyperspectral imaging. Avocados are produced every week of the year. At the packing house, avocados with different dry matter contents are mixed. The ability to classify dry matter noninvasively using hyperspectral technology has the potential to create significant economic value. It also can reduce waste at the restaurant level because it gives food service companies the ability to ripen avocados in select bands to better optimize distribution in accordance with ready-to-eat quality.
Hyperspectral data can enable greater information sharing all across the value chain for numerous benefits, delivering greater security and transparency, and playing a role in building the food system of the future.
Meet the authors
Abi Ramanan is the CEO of ImpactVision, a software company combining hyperspectral imaging with machine learning for food companies’ supply chains to reduce waste and fraud and maximize the yield and consistency of products delivered to consumers; email: [email protected].
Gustav Nipe is the CTO of ImpactVision and the architect of the company’s hyperspectral imaging classification system; email: [email protected].
Spectroscopy and Hyperspectral Imaging
Spectroscopy devices generate a spectral plot at a specific point, providing only spectral information. Hyperspectral imaging cameras produce a spectrum at each pixel in an image by integrating computer vision techniques. We can think of hyperspectral images as using a spectrometer for each pixel in an image resulting in acquisition of both spectral and spatial information continuously. Traditional spectroscopy can utilize absorption, reflectance, or fluorescence, but hyperspectral imaging typically involves reflectance.
A single-point analysis of meat, for example, may result in a lean pixel or a fat pixel, but that says little about the overall distribution of fat across the sample. With a hyperspectral image, thousands of pixels are analyzed simultaneously, providing information about more complex parameters, such as intramuscular fat or tenderness. Similarly, assessment of a single pixel on the skin of a fish cannot provide a strong indication of the overall state of decay, which is required for assessing and grading freshness.