Food sorting traces its origins all the way back to the nomadic hunter-gatherer days. While foraging out in the wild, gatherers handpicked fruits, vegetables, seeds, and nuts that would be shared with the rest of the tribe. Lacking the chemical analysis and knowledge that are available today, a tribe discovered what was and was not safe to eat through trial and error. 3D laser profilers can ensure that each cookie looks like the one shown on the package. Courtesy of Teledyne DALSA. Those early humans were not as picky as we are today, but everything they ate was likely given at least a brief once-over. By the 20th century, food inspection was conducted much faster. Workers stood along conveyor belts and manually sorted products that zipped past. Accurate sorting at high speeds and high volumes required unwavering focus, consistently high attention to detail, and endurance. Humans are prone to fatigue, distractions, and mistakes. To make up for these human shortcomings, the first mechanical, automated optical sorters were developed in the 1930s and were quickly adopted for bean sorting. The benefits of such systems were immediately obvious. Since then, the technology has been applied to the sorting of fruits, vegetables, seeds, nuts, meat, coffee, tea, and grains. Apples are sorted by lane sorters, where 2D cameras inspect the fruit. Courtesy of Teledyne DALSA. The deeper into the infrared a camera looks, the deeper into the flesh of the apple it can see to spot early signs of bruising. MNF: minimum noise fraction. Courtesy of Teledyne DALSA. Going beyond agricultural products, the same optical sorting methods were used for metals, minerals, plastics, and waste. In a modern color sorter, cameras (or sensors) replace human eyes, and paddles or precise streams of air replace human hands. As a result, these systems can operate at higher speeds and process higher volumes more accurately, more consistently, and for longer periods of time than humans ever could. Picture-perfect produce Color sorting systems do not depend just on color when making decisions. They need to take other factors into account as well. A tomato that has been crushed during transport is still red, but it is no longer sellable as a whole tomato. In addition to color, optical sorters recognize size and shape. The operator or systems integrator can set the tolerances to establish whether a tomato conforms to a certain standard. Too big, too small, or crushed tomatoes — or ones that have the correct color but the wrong shape — and so on, can be removed so that only picture-perfect produce makes it through to the other side. SWIR cameras are typically sensitive from 900 to 1700 nm, though sensitivity up to 2200 nm is possible as well. Courtesy of Teledyne DALSA. A laboratory image shows a situation in which color and shape may be insufficient to distinguish between desirable and undesirable materials (left). SWIR detects moisture as an additional layer of information (right). Courtesy of Teledyne DALSA. Tomatoes that fall short of being picture-perfect may not make the cut for the grocery store but may still be used elsewhere. This is why an optical sorter does not sort in terms of good or bad product but based on a continuum. Pieces of debris and rotten fruit deemed unacceptable in this context are disposed of. An ugly tomato, however, still has value and can be sorted into a separate bin from its more attractive peers — later to be canned, crushed, or juiced. The same concept applies to frozen fruits, pie fillings, juices, and preserves. Color sorting systems can do everything that human sorters can do but faster, more reliably, and on a much larger scale. However, like the human eye, these sorters are based only on what can be seen in the visible spectrum: shape and color. This includes surface defects, as well as contaminants — such as a piece of wood, which is relatively easy to detect among cherries on a conveyor. Defects and contaminants are not always visible. The same piece of wood can effectively disappear when a system is sorting nuts that are similar in color to the wood. A piece of fruit may be bruised or rotting below the surface. And metal, bone, or plastic may be embedded inside a piece of meat. If a contaminant cannot be seen, it cannot be detected by a color sorter. From NIR to SWIR Enter nonvisible techniques. The absorption coefficient of water has a few peaks, including one at 970 nm (near-infrared, or NIR) and another, higher peak at 1440 nm (shortwave infrared, or SWIR). This means that these wavelengths do not get very far when they meet water — they are absorbed. As a result, NIR and especially SWIR wavelengths are excellent at detecting moisture because moisture shows up very dark on an IR imager, in contrast to the scene. The presence of moisture can indicate bruising, even if a bruise is not yet fully formed and is below the surface. The absence of moisture can indicate that something does not belong, such as a piece of glass. An infrared camera is quite useful in sorting applications because it can uncover details that cannot be distinguished with visible light. Applications for infrared imagers extend far beyond inspecting produce. Eggs are rinsed before packaging. Machine vision helps in the grading and quality inspection of eggs. Courtesy of Teledyne DALSA. Eggs are rinsed before packaging. Machine vision helps in the grading and quality inspection of eggs. Courtesy of Teledyne DALSA. A color camera typically has red, green, and blue filters deposited on the sensor. These filters each cover a range of light in the visible spectrum. Infrared sensors cover a different range of light. Both cases involve only a few, relatively broad spectral ranges, or bands. Hyperspectral imagers, however, can detect hundreds of narrow bands. Like water, other chemicals absorb and reflect light differently at different wavelengths. As a result, some chemicals and objects have unique spectral signatures that can be used to identify them. The more spectral bands that are collected, the more materials that can be identified, and thus, the more precisely they can be sorted. This is sometimes called chemical imaging. And, with it, users can detect parameters that affect flavor — moisture, acidity, sugar, and more. With advanced systems, it is possible to see whether a piece of fruit will be sweet or not. Hyperspectral imaging can also aid in the detection of bruising, chilling injury, firmness, ripeness, mineral content, soluble solid content, fruit fly infestation, bacterial diseases, and even fecal matter. Perhaps most importantly, such detection is all done in a nondestructive way, further reducing waste. Many choices Going beyond visible, nonvisible, and hyperspectral imaging, a 3D solution can be used to inspect cookies and cakes for the correct size and volume, so that each cookie is consistent and fits into the packaging. UV light can aid in detecting mold and aflatoxin. X-rays and metal detectors are widely used in meat processing to catch metal and bone. Raisins that have been washed and sorted are ready for packaging. Machine vision can also be used to detect irregular pasta (top) and the sugar ends defect that can afflict potatoes (bottom). Courtesy of Teledyne DALSA. Each inspection modality for food and beverage products brings a different capability that is well suited for capturing issues that the others cannot. Therefore, the more modalities that are installed, the better the defect detection will be. With better defect and contaminant detection, the quality of the output increases, and with more accurate sorting, the amount of waste decreases. Deploying all of the above technologies is not necessarily feasible, however. A number of factors, including cost and complexity, must be considered in the system design. One factor to consider is whether the food being inspected has been presorted in any way. Presorting by weight, for example, may eliminate the need for 3D volumetric measurements and also may eliminate certain contaminants without using any vision at all. Also, a lot of debris can be shaken or rinsed off. Much depends on where in the process inspection takes place, and on the end goal. For example, after less desirable potatoes have been separated out and sliced to make french fries, sorting by color is enough to guide the system to separate brown parts from the yellow or white flesh. Some potatoes are afflicted by an invisible and undesirable defect called sugar ends, localized to one end or the other of the potato. This problem can be solved by cutting off both ends of the potatoes (simple but wasteful), or by using hyperspectral imaging to detect the affliction and then cutting only where required (a more expensive solution that leads to higher quality and higher yields). The matter of system cost must be considered as well. Though advancements continue, a color camera is generally less expensive than a SWIR camera, which is less expensive than a hyperspectral camera. Determining the best option often comes down to the purity of the input, the quality requirements of the finished output, and the value of the good itself. Sorting an inexpensive and plentiful item such as rice may not justify the cost of a hyperspectral system, whereas sorting tea leaves might. Also, while a line-scan camera is well suited for use in seed sorting, this type of camera may not fit quite as well in an application requiring whole-fruit imaging, where an area-scan camera is more suitable. The different technologies are suited for different situations. When considering the optimal inspection solution for a food or beverage product, much work has already been done to record the spectral characteristics of various materials, so a particular defect or contaminant may have already been categorized. However, trial and error are often the best way to find a solution. A knowledgeable supplier, distributor, or systems integrator should be able to help. Meet the author Mike Grodzki is a product manager for Teledyne DALSA’s machine vision products. He holds an MBA from Wilfrid Laurier University and has a keen interest in machine vision as a tool for lessening our impact on the environment; email: mike.grodzki@teledyne.com.