Hyperspectral and multispectral imaging are increasingly beneficial for a range of applications as diverse as agriculture, health care, and remote sensing.
JAYLOND COTTEN-MARTIN, EDMUND OPTICS INC.
Over the past two decades, hyperspectral imaging (HSI) and
multispectral imaging (MSI) have been growing in prominence and utility.
Although the terms are often conflated, they represent two distinct imaging practices, each fitting its own application spaces. Both technologies present advantages over standard machine vision
imaging, which uses light only from the visible spectrum. However, with the benefits of HSI and MSI comes increased complexity of the imaging systems in terms of lighting, filtering, and optical designs.
Hyperspectral and multispectral imaging are used in agriculture to monitor the health of fields across a broad range of the electromagnetic spectrum.
In typical machine vision applications,
the illumination used and captured by the sensor is from roughly 400 nm (violet)
to 700 nm (dark red) (Figure 1). The light is often collected with imaging lens assemblies and sensors that have peak spectral sensitivities around 550 nm.
The quantum efficiency of most camera sensors, or the ability to convert photons into an electric signal, decreases significantly when extended into the UV or the NIR.
Figure 1. Wavelength regions outside of the visible spectrum are used in hyperspectral and multispectral imaging. Courtesy of Edmund Optics.
HSI, in simplest terms, captures images that contain information from a broader portion of the electromagnetic spectrum. This range can start with UV light,
ex-tend through the visible spectrum,
and end in the near-IR or shortwave IR. The extended wavelength range can reveal properties of material composition that are not otherwise apparent.
A sensor used in machine vision will output an array of grayscale values that result in a 2D image of some object within a viewing area. The functional utility of this is generally feature recognition for the purposes of sorting, measuring, or locating objects. Unless optical filtering is used, the vision system is unaware of the wavelengths that are being used for illumination. This is obviously not true for sensors that have a Bayer filter pattern (RGB), but even then, each pixel is restricted to accepting light from a narrow band of wavelengths, and the camera software assigns color after the fact. In a truly hyperspectral image, each pixel corresponds to coordinates, signal intensity, and wavelength information. For this reason, HSI is often referred to as imaging spectroscopy1.
As a quick aside, spectrometers collect wavelength information as well as relative intensity information for the various wavelengths detected2. These devices typically collect light from a singular source or location on a sample. They can be used to detect the presence of substances that scatter and reflect specific wavelengths, or material composition based on fluorescent or phosphorescent emissions. An HSI system takes this technology to the next level by assigning positional data to the collected light spectrums. A hyperspectral system does not output a 2D image, but instead a hyperspectral data cube or image cube3.
Image acquisition modes
Four primary hyperspectral acquisition modes or methods are used today, each with its own advantages and disadvantages (Figure 2). The first is called whiskbroom and is a point scanning process that acquires the spectral information for one spatial coordinate at a time. This method tends to offer the highest level of spectral resolution, but requires the system to scan the target area in both the x- and y-axes, adding significantly to the total acquisition time1.
Figure 2. The four primary hyperspectral acquisition modes are point scanning, or whiskbroom (a); line scanning, or pushbroom (b); plane or area scanning (c); and single shot, or snapshot (d). Courtesy of Edmund Optics.
The second mode, called pushbroom,
uses a line-scanning data-capture technique. Only a single axis of spatial
movement is required while a row of pixels scans over an area to capture the spectral and positional information.
Pushbroom systems can have compact size, low weight, simpler operation, and higher signal-to-noise ratio1. When using this method, it is critical to time the exposures just right. Neglecting to do so will introduce inconsistencies regarding saturation or underexposure of spectral bands.
Some believe MSI is just a lower form of HSI with reduced spectral resolution. In truth, the two technologies each present advantages depending on the task.
The third method is plane scanning. This mode images the entire 2D area at once, but at each wavelength interval.
The process involves capturing numerous
images to create the spectral depth of
the hyperspectral data cube. While this
capture method does not require translation of the sensor or full system, it is critical that the subject remain still during acquisition; otherwise, the accuracy of positional and spectral information will be compromised.
The fourth and most recently developed method is referred to as single shot or snapshot. A single-shot imager collects the entirety of the hyperspectral data cube within a single integration period1. Although single shot appears to be the preferred future of HSI implementation, it is currently limited by comparatively lower spatial resolution and requires further development1.
Key differences
An MSI system is similar in many ways to an HSI system, with key differences. In comparison to data collection using effectively continuous wavelengths with an HSI system, MSI concentrates on several wavebands that are preselected based on the application. Common RGB sensors help illustrate the differences between HSI and MSI systems. In an RGB sensor, a Bayer pattern — consisting of red, green, and blue filters — is layered over the pixels. The filters allow wavelengths from specific color bands to be absorbed by the pixels while the rest of the light is attenuated. These bandpass filters have transmission bands in the range of 100 to 150 nm and have slight spectral overlap (Figure 3). The images captured are then rendered with false color to approximate what the human eye sees. In most MSI applications, the wavelength bands are significantly narrower and more numerous. The wavebands are commonly on the order of tens of nanometers and are not exclusively a part of the visible spectrum. Depending on the application, UV, NIR, and thermal wavelengths (midwave IR) can have isolated channels as well4.
Figure 3. A quantum efficiency curve for an RGB camera showing the overlap between red, green, and blue. Courtesy of Edmund Optics.
Some believe MSI is just a lower form of HSI with reduced spectral resolution. In truth, the two technologies each present advantages, depending on the task. HSI is best suited for applications that are sensitive to subtle differences in signal along a continuous spectrum. These small signals could be missed by a system that is sampling larger wavebands. However, some systems require that significant portions of the electromagnetic spectrum be blocked and that light be only selectively captured (Figure 4). The other wavelengths could present significant noise that could potentially ruin measurements and observations. Also, if less spectral information is included in the data cube, the image capture, processing, and analysis can happen more quickly.
Figure 4. Multi- and hyperspectral stacks of images. Comparison of the image stacks in MSI, in which several images are taken across various discrete wavelength regions, and HSI, in which images are taken over a larger continuous range of wavelengths. Courtesy of Edmund Optics.
Applications
The number of applications that require HSI and MSI continue to grow. These technologies have large footprints in the life sciences and remote sensing. More specific markets include agriculture, food
quality and safety, pharmaceuticals, and health care3. Farmers find both technologies particularly useful. To scan over fields, tractors and drones can be equipped with spectral imagers. By analyzing spectral characteristics of the captured images, farmers can accurately determine the status of crops, including the general health of plants, the state of the soil, regions that have been treated with
certain chemicals, or the presence of something harmful, such as an infestation. All the information has unique spectral markers that can be captured, analyzed, and used to ensure the optimal production of produce.
The same is true in health care. With the help of hyperspectral imagers, doctors can now conduct noninvasive scans of skin to detect diseased or malignant cells. Certain wavelengths are better suited for penetrating deeper into the skin, so a more detailed understanding of a patient’s condition can be reached. Given the correct stimulation, cancers and other diseased cells will fluoresce or absorb light, making them easily distinguishable from healthy tissue. Doctors are no longer required to make educated guesses based on observations or a patient’s description of symptoms. Sophisticated systems can record and automatically interpret the spectral data. This can lead to significantly expedited diagnoses and rapid treatment of the exact areas affected5.
For decades, HSI and MSI have been crucial components of remote sensing, which involves aerial imaging of Earth’s surface with the use of unmanned aerial vehicles (UAVs) and satellites. Spectral photography can penetrate through Earth’s atmosphere and various types of cloud cover for an unobscured view of the ground below. Remote sensing can be used to monitor changes in population, observe geological transformations, and study archeological sites, among other tasks. These imaging techniques have become increasingly critical in the study of the environment. They enable data to be collected on deforestation, ecosystem degradation, carbon recycling, and erratic weather patterns. Researchers use the information gathered to create predictive models of the global ecology. These models drive many environmental initiatives meant to combat the negative effects of climate change and human influence on nature6.
Barriers to adoption
The number of application spaces that benefit from HSI and MSI is large and increasing, but complexities in the current technology can slow adoption in some industries. The systems are significantly more expensive when compared to other machine vision components. The sensors are more complex, have broader spectral sensitivity, and must be precisely calibrated. The sensor chips often require the use of substrates other than silicon, which is sensitive only from approximately 200 to 1000 nm. The materials indium arsenide (InAs), gallium arsenide (GaAs), and indium gallium arsenide (InGaAs) are used for collecting light up to 2600 nm. If the requirement is to image from the NIR through the MWIR, a mercury cadmium tellurium (MCT, or HgCdTe) sensor is required. The sensors and pixels used in these systems will also need to be larger than many machine vision sensors to attain the required sensitivity and spatial resolution1.
Another challenge exists in pairing these high-end sensors with the proper optical components. Bandpass filters, diffractive optics such as prisms or gratings, and even liquid crystal or acousto-optic tunable filters are required to separate light of differing wavelengths so the spectral data can be recorded7. Additionally, the lenses used for these cameras must be designed to work optimally across vast wavelength ranges and temperature fluctuations. The need for more optical elements in the designs increases system cost and weight. These elements will also need to have different refractive indices and dispersive properties for broadband color correction. With differing glass types comes varied thermal and mechanical properties as well. After selecting glasses that have the appropriate internal transmission spectra, broadband multilayer antireflection coatings must be applied to each lens to ensure maximum light throughput. The multitude of unique requirements in these circumstances makes for a tedious process of lens design for HSI and MSI, and such design requires great skill. Certain application spaces also necessitate that the lens assemblies are athermal to ensure that a system will function the same whether used on the ground or in the upper atmosphere.
The goals of future development include making HSI and MSI systems more compact, affordable, and user-friendly. Improvements in these areas will encourage new markets to use the technology and will advance the markets that already do.
Meet the author
Jaylond Cotten-Martin is a vision solutions engineer for Edmund Optics Inc. He is responsible for consultation on high-level imaging and machine vision applications. He received a B.S. in physics from The College of New Jersey and is an AIA Certified Vision Professional.
References
1. T. Adão et al. (2017). Hyperspectral imaging: a review on UAV-based sensors, data processing and applications for agriculture and forestry. Remote Sens, Vol. 9, Issue 11, p. 1110, www.doi.org/10.3390/rs9111110.
2. D.W. Ball (2006). Field Guide to Spectroscopy. Bellingham, Wash.: SPIE Press.
3. F. Vasefi et al. (2016). Hyperspectral and multispectral imaging in dermatology. In Imaging in Dermatology. M.R. Hamblin et al., eds. Amsterdam: Elsevier Science.
4. R. Paschotta (2008). Multispectral imaging. In Encyclopedia of Laser Physics and Technology. Vol. 1. Weinheim, Germany: Wiley-VCH.
5. A. Schneider and H. Feußner (2017).
Biomedical Engineering in Gastrointestinal Surgery. Cambridge, Mass.: Academic Press.
6. S. Unninayar and L. Olsen (2008). Monitoring, observations, and remote sensing — global dimensions. In Encyclopedia of Ecology. S.E. Jørgensen and B.D. Fath, eds. Amsterdam: Elsevier Science, pp. 2425-2446, www.doi.org/10.1016/B978-008045405-4.00749-7.
7. I. Schelkanova et al. (2015). Early optical diagnosis of pressure ulcers. In Biophotonics for Medical Applications. I. Meglinski, ed. Cambridge, England: Woodhead Publishing, pp. 347-375, www.doi.org/10.1016/b978-0-85709-662-3.00013-0.