Bert Geelen, Nicolaas Tack and Andy Lambrechts, imec
A new snapshot multispectral imager enables real-time, dynamic applications; its sister line-scan hyperspectral imager handles applications demanding high speed and high spatial/spectral resolution. Both solutions use monolithically integrated Fabry-Perot filters on top of an image sensor for low cost, compactness and high speed.
By combining spectroscopy and imaging, multi- and hyperspectral cameras allow us to identify or differentiate spectral signatures of various materials and objects. They “see the unseen” by splitting the light reflected by an object into many spectral bands, which they capture and process separately. That way, for each pixel in a scene, a spectral signature is recorded.
While hyperspectral imaging deals with capturing many narrow bands over a continuous spectral range, multispectral imaging senses either a lower or a discrete number of bands. Both spectral imaging techniques organize their information into three-dimensional spectral data cubes. These cubes have one spectral and two spatial dimensions, and can be interpreted as a stack of images, one for each wavelength.
The potential of spectral imaging is enormous. Although originally applied in geology and military surveillance, engineers have worked hard to broaden the application domain toward food inspection, civilian security and surveillance, precision agriculture and livestock farming, and even medical applications.
The requirements of all these intended applications define wide-ranging specifications, implying trade-offs between spatial, spectral and temporal resolution. But all applications have a few requirements in common: low cost, small size and high speed. In current tools, the presence of fore-optics like prisms, lenses and slits makes them bulky and expensive, with prices of typically in the tens of thousands of euros. Also, many tools are slow in operation: Scanning the objects takes several seconds, or even minutes. For these reasons, the technique so far has not found widespread industrial acceptance.
A practical solution for fast, compact, user-friendly and low-cost spectral imaging cameras consists in monolithically integrating optical interference filters on top of CMOS-based image sensors. These integrated filters can be made by traditional semiconductor fabrication tools, enabling mass production and, therefore, low-cost manufacturing. This unique approach, developed by imec, has been applied in two different camera architectures: a snapshot multispectral imager and a line-scan hyperspectral imager. Both are fast and compact, but they differ in spatial, spectral and temporal resolution. Therefore, they generate a complementary set of vision application opportunities.
Integrated hyperspectral filters
A key concept of these spectral imagers is the use of Fabry-Perot filters that are directly postprocessed at wafer level on top of a CMOS image sensor. A Fabry-Perot filter consists of two tightly spaced parallel mirror surfaces, between which incoming light reflects and interferes. This interference allows some wavelengths to pass, while others are filtered out. By carefully controlling the distance between the reflecting surfaces, a filter can be designed to transmit only a narrow-wavelength band. The design of the filter structures requires only CMOS-compatible materials and suitable fabrication steps. As such, they can be processed directly on top of the wafer that contains the image sensor chips (Figure 1).
Figure 1. Wafer with integrated Fabry-Perot filters.
Most spectral imagers use dispersive optics to extract spectral information, and the few solutions that use comparable filter structures produce them separately from the imager. This new integration approach is therefore unique and has important advantages: First, combining the production of filters and sensors into one CMOS-compatible flow leads to an overall simplification and cost reduction, and enables mass production. Second, the monolithic integration induces less crosstalk between neighboring pixels and reduces stray light in the system.
This has a positive effect on the system’s sensitivity and speed. It should, however, be noted that the process steps and materials used for postprocessing the filter structures must be carefully selected. They are crucial for the quality of the Fabry-Perot filters but may not damage the underlying CMOS sensor.
Snapshot imager with high temporal resolution
An innovative snapshot multispectral imager, based on the above concept, allows an entire image to be captured at one discrete point in time. This is realized by adding three key steps to the use of monolithically integrated filters: (1) organizing the filters in a tiled configuration, where each filter is designed to sense only one narrow band of wavelengths; (2) adding an optical subsystem that duplicates the scene onto each filter tile; and (3) using an objective lens that forms an image of the scene on the optical duplicator.
Figure 2. (top) Spectral cube with number of bands (NB) and spatial resolution of W 3 H mapped to NB tiles on the 2-D sensor; (bottom) schematic representation of an integrated implementation of the Fabry-Perot filter on top of a CMOS imager in a tiled layout. FSI 5 front-side illuminated.
These building blocks allow duplicating each spatial point of the scene multiple times on the sensor array, such that the different spectral samples are sensed simultaneously. In other words, all three dimensions of the spectral cube (two spatial and one spectral) are mapped on the two dimensions of the sensor. This way, the full spectral cube is acquired in one frame period, enabling true snapshot spectral imaging (Figure 2).
Based on this principle, we built a prototype camera (Figure 3), processing 32 spectral filters onto a 2-megapixel sensor. The filters cover part of the VNIR (visible and near-infrared) range for which CMOS imagers are sensitive (600-1000 nm). Each tile senses one selected wavelength by varying the cavity length for each filter. This is achieved by using a set of CMOS-compatible production steps such as deposition, patterning and etching. The optical duplicator subsystem is made of off-the-shelf components consisting of a microlens array and optomechanical components for assembly. The duplicator relays the light of the objective lens onto the sensor through multiple optical channels, each consisting of one lens or lens system.
Figure 3. The multispectral snapshot camera consists of a replaceable objective lens at the front and duplication optics between the objective and the camera, which relay and copy the scene as seen through the objective lens onto the tiled filters of the sensor inside the camera at the back.
A field stop is located at the image plane of the objective lens to maximize the spatial resolution: Each image copy shows the same part of the scene and completely occupies one filter tile without overlap between neighboring image copies. The resulting prototype camera acquires the multispectral image cubes of 256 x 256 pixels at a speed of about 30 frames per second in daylight conditions. At higher illumination levels, as typically used in machine vision applications, a speed of 340 frames per second can be achieved (Figure 4).
Monolithic integration of the dedicated spectral filters at its core enables the use of mass-produced fore-optics, reducing total system cost while retaining high-quality spectral acquisition characteristics. The prototype snapshot camera has a high temporal resolution and can therefore be deployed in real-time, high-speed, dynamic applications. This is facilitated by mapping the entire spectral cube simultaneously onto the sensor. In scanning-based systems, the cube is scanned over time by successively mapping a subset of the cube onto the sensor. This means that scanning-based systems can generate cubes with higher spatial and spectral resolution. Therefore, we call our snapshot camera “multispectral,” even though it can sense a larger number of bands compared to many other multispectral systems. Moreover, thanks to an optimized optical duplicator, the sensor usage and cube resolution are maximized.
Figure 4. The multispectral snapshot camera at the top acquiring a scene of a cactus on a dark coaster, for which the spectral content acquired in real time is shown on the screen at the back.
Line-scan hyperspectral imager
The prototype snapshot camera builds on the expertise gained with an earlier and more mature spectral imager solution that requires scanning the scene. This line-scan imager also makes use of monolithically integrated optical interference filters, and therefore holds the same benefits in terms of cost, size and speed. In this imager, the filters are not organized in a tiled structure but as a stepwise wedge on top of the CMOS sensor. The line filters for the different wavelengths are built by varying the cavity length in one direction over the sensor while keeping it fixed over the second dimension. As the imager scans 100 spectral bands in the 600- to 1000-nm wavelength range, the solution is truly “hyperspectral” (Figure 5).
The speed of the line scanner corresponds to 2880 lines per second, so in terms of speed, the imager therefore exceeds current state-of-the-art hyperspectral line scanners. These scanners typically scan the image line-per-line and collect all wavelengths for that single line. Our camera can cover multiple spatial lines in one spectral band, thus enabling parallel acquisition of these lines. This is an important reason for the speedup in our camera. For example, implementing the filter on a sensor with 4 megapixels and 128 bands has 16 lines under a filter for one specific spectral band. The line rate for that imager is thus 16 times the frame rate of 180 frames per second – or 2880 lines per second.
Figure 5. Imec’s hyperspectral image sensor mounted on a development board.
The line scanner is especially suited for applications that demand low cost, high speed and compactness in combination with high spatial/spectral resolution. Moreover, this hyperspectral imager can, by design, be flexibly tuned to multiple applications. Currently, most hyperspectral cameras analyze specific problems in lab environments. For a particular problem, researchers use a dedicated camera with a reduced number of spectral bands. Because of its high spatial and spectral resolution, this new camera offers more flexibility and is therefore more suited for industrial applications.
For example, in food inspection, a camera can be used for inspecting carrots, and, by adjusting the software, the same camera can be deployed the next day for inspecting beans. The line-scanner solution is now available for evaluation by interested companies.
Myriad applications
While so far used mainly in research environments, the technology now has been tuned to more industrial needs. This will have an impact in several application areas.
The line scanner will be suited for applications that demand a high spatial and spectral resolution. Think of the level of flexibility that is required in precision agriculture, where the farmer can use the tool to inspect his land and determine, for example, the best places to selectively spray pesticides on his crops. Other application domains include Earth observation (e.g., water quality monitoring), food inspection and sorting, industrial process control (e.g., fuel quality monitoring) and even medical applications (e.g., skin health monitoring). The technology is versatile and can easily fit multiple application needs. Still, the camera must scan the image and is therefore not ideal for real-time, dynamic applications.
Complementary to these applications, the snapshot camera opens opportunities in video-rate and nonscanning application domains. The high temporal resolution makes it attractive for analyzing dynamic scenes in security applications such as detecting license plates of driving cars, or tracing intruders in or around buildings. Also in farming, the snapshot technique can be deployed to capture dynamic scenes and is as such not restricted to precision agriculture. For example, the tool can monitor and recognize individual animals to allow precision livestock farming. In machine vision or industrial process control, it is possible to monitor the progress of irreversible processes such as coffee bean roasting. The snapshot camera also can be used to analyze nondynamic scenes. Since no additional setup is required for scanning, it can easily be deployed as a portable system. For example, it could be used for colorimetry applications in the field, such as analyzing the color of paintings in museums.
Meet the authors
Bert Geelen is a member of the Integrated Imaging team at imec in Leuven, Belgium; email: [email protected]. Nicolaas Tack, a researcher, is also an Integrated Imaging team member; email: [email protected]. Andy Lambrechts leads the imec Integrated Imaging team; email: [email protected].