The applications, advantages and impact of microsized HSI enable compact, high-end and even autonomous intelligent cameras. Hyperspectral imaging (HSI) has long been confined to laboratory use and expensive applications. This is changing now, thanks to the advent of fast, compact, user-friendly and low-cost hyperspectral cameras based on a revolutionary hyperspectral sensor monolithically integrated on a CMOS imager. The advantages and versatility of microsized HSI will positively impact several application areas, including agricultural crop inspection. Miniaturization and lower cost also make it ideal for point-of-care medical appliances; for example, researchers are currently designing a device for use in ophthalmology. In the future, HSI technology will cross into consumer appliances and devices, such as mobile phones, where an integrated hyperspectral sensor can be used to analyze food, allowing users to measure caloric intake or detect bacterial contamination. Wafers with various hyperspectral filters postprocessed on top of CMOS imagers enable a variety of applications for hyperspectral imaging. Photo courtesy of imec. Several vendors have already built compact cameras equipped with imec’s first HSI sensors, leading to high-end cameras for industrial inspection, as well as extremely compact or autonomous intelligent cameras. Additionally, prototype sensors with various specifications could lead to the fabrication of custom sensors and software geared toward specific applications that extend the spectral range. HSI on a single chip Traditionally, hyperspectral cameras have been assembled by hand from a high number of discrete components, including expensive and heavy glass precision optics. Assembly requires extremely careful alignment and calibration. These cameras typically carry price tags in the tens of thousands of dollars. Despite the expense, many tools are slow in operation; scanning objects or scenes still takes several seconds or even minutes. To produce hyperspectral cameras small enough for integration with devices such as point-of-care medical instruments, state-of-the-art HSI technology must advance by orders of magnitude in size (reducing to a form factor of cubic centimeters or even millimeters), cost (from tens of thousands of dollars to a few dollars per system) and complexity (everyone should be able to use them). This would be a major leap, possible only if as much hyperspectral functionality as possible could be integrated on a CMOS chip, which could then be manufactured with standard processes in a high-volume fab. One obvious approach is to replace most of the optical elements needed to filter light into the many spectral bands with integrated thin-film structures. This is what imec set out to do some five years ago. The idea was to postprocess hyperspectral filters on top of a high-quality commercial image sensor, directly at the wafer level, with only CMOS-compatible materials and fabrication steps. From left to right: an HSI chip with 32 filter tiles, an optical duplicator and an HSI camera with integrated tile-chip and duplicator. Photo courtesy of imec. Compared to assembled cameras, such monolithic integration eliminates unwanted reflections between different discrete components and reduces stray light in the system. It also eliminates a host of error-prone and labor-intensive mounting procedures, replacing them with repeatable, nanometer-precision processes. The ability to mass-produce would offer a major advantage, incorporating hyperspectral functionality into the existing manufacturing infrastructure for image sensors, including assembly, testing, packaging and calibration. HSI line-scanning filter For our first hyperspectral chip, imec looked to the concept of Fabry-Perot filters to replace diffraction grating that, in traditional HSI cameras, splits the light into different wavelengths. An array of Fabry-Perot filters, each covering a specific small wavelength band, enables coverage of a whole spectrum with small adjacent bands. This array takes the form of a so-called Fabry-Perot staircase. Each step, a few micrometers wide and perfectly aligned to the pixels below, is one filter. The incoming light interferes between the top surface of each step and the bottom surface of the staircase. As a result, the step transmits only one spectral band to the image sensor just below it. A postprocessed Fabry-Perot staircase (not to scale) enables whole-spectrum coverage with small adjacent bands. Photo courtesy of imec. The resulting hyperspectral filter is directly postprocessed at wafer level on top of a commercially available CMOSIS CMV2000 image sensor, which is a 2-MP sensor with maximum frame rate of 340 fps. The hyperspectral filter has 100 spectral bands between 600 and 1000 nm. The filter bandwidth is between 4 and 10 nm, depending on the lens that is used. The transmission efficiency of the filters is around 85 percent. The hyperspectral line-scan imager captures wedges of 8 pixels. This makes them fast. Moreover, the spatial information within one band of 8 pixels enables the device to compensate for uncontrolled movements. The resulting system may scan up to 2880 lines per second. This significantly exceeds available state-of-the-art laboratory equipment and is compatible with the requirements of high-speed industrial and medical applications. The design and fabrication process is very flexible. If an application requires extreme sensitivity, the staircase design can be adapted for more steps in that range, covering narrower bands. Alternatively, if only a smaller number of well-defined bands is needed, a filter can be designed to match the requirements perfectly. From hyperspectral picture to hyperspectral movie Line scanners are ill-suited for 2-D scenes in which the camera or scene must move freely. Think of an inspection camera in a probe robot, a free-flying drone, a movement-activated security camera or a snapshot taken with a smartphone. These require whole-frame acquisition, wherein an entire three-dimensional, multispectral data cube is sensed at once. For this kind of application, the Fabry-Perot filters can be reorganized in a tiled configuration, with each tile responsible for sensing one narrow band of the spectrum for the whole scene. On a 2-MP sensor, it is possible to fit 32 tiles, each with a resolution of 256 × 256 pixels. Adding an optical duplicator between the objective lens and the filter enables 32 identical copies of the scene on the tiles to create a hyperspectral snapshot of one scene. The resulting prototype camera acquires multispectral image cubes of 256 × 256 pixels at a speed of about 30 fps in daylight conditions. At higher illumination levels, typically used in machine vision applications, it can achieve a speed of 340 fps. This is the limit of the image sensor. The 32 tiles scan 32 bands in the visible and near-infrared, between 600 and 1000 nm, with 12-nm incremental steps. The spatial resolution of every tile can be increased either by moving to a larger sensor or by reducing the number of spectral bands. This camera allows real-time hyperspectral imaging at video rates. It captures all spectral bands in parallel and possesses high transmission efficiency. All of this is made possible in a simple design with potential for low-cost, high-volume production, because the optical duplicator is a non-wavelength-specific component, while the very sensitive, wavelength-specific functionality is integrated at wafer level. Additionally, the technology holds the potential to tune and adapt the design, and to choose the spectral bands best suited to optimize the application’s discrimination power. Hyperspectral mosaic filter Going one step farther, the imec team developed a mosaic filter, a filter type that is also used in traditional imagers to obtain red, green and blue images. The pixels are grouped into 3 × 3 or 4 × 4 arrays. On top of each such group of 9 or 16 pixels, 9 or 16 narrowband filters are processed, effectively making filters the size of one imager pixel. With this mosaic design, there is no longer a need for separate duplicator hardware. An HSI chip with per-pixel filters forming a 4 × 4 pixel mosaic. Photo courtesy of imec. To obtain a full image with good spatial resolution, various demosaic algorithms can be used to interpolate a complete set of values for each pixel. This design enabled a first prototype with 4 × 4 pixels and scanning 16 bands between 465 and 630 nm in 11-nm incremental steps. Spatial resolution may vary between 512 × 272 pixels and 2 MP, depending on the demosaic algorithm used. Building the cameras The ability to fabricate hyperspectral functionality on a chip – while ensuring it will be extremely compact, reliable and cost-efficient – is essential for the introduction of HSI to new markets. A number of camera builders have already integrated the chips into compact HSI cameras, from a high-end industrial inspection unit to an extremely compact USB3 camera. Ximea GmbH, a developer of machine vision, industrial and scientific cameras, has integrated the hyperspectral sensors into its ultrasmall, lightweight xiQ cameras. With respective weights of 27 g, dimensions of 26.4 × 26.4 × 21.6 mm, and power use of 1.8 W, these are the world’s smallest industrial USB3 HSI cameras. This makes them useful in small unmanned aerial vehicles (UAVs) and drones, where power use and thermal management are key, and payload weights critical. Such drones are in high demand in the emerging field of precision farming. This ultracompact HSI camera from Ximea is designed for industrial uses including UAVs and drones. Photo courtesy of Ximea. BaySpec Inc. offers a handheld Android device on which customers can write their own analysis and visualization apps. With a form factor resembling that of a mobile phone, this camera can be used for inspection on the factory floor, in the field or in remote areas. 3D-One BV has developed a number of flexible camera modules that OEMs can combine into end-user systems, such as sorting equipment. Tattile Srl makes smart cameras for industrial and machine vision¬¬, including a reprogrammable FPGA that can be installed directly onto a conveyor belt and allows all processing within the camera¬¬. For one example, these cameras are suited to food-sorting lines, where cameras monitor direct sorting processes. VRmagic AG focuses on the UAV market, remote sensing and machine vision; its cameras combine compact size with built-in corrections and storage. And a final camera builder, Adimec, delivers high-end cameras for applications where reliability, robustness and high speed are critical. Extending range and applications Spectral imaging was originally used in geology and military surveillance, and engineers have worked hard to broaden its application domain to include food inspection, civilian security and surveillance, precision agriculture, pharmaceutical inspection and even medical applications. Until now, the hurdles to widespread use were the complexity, cost and size of the cameras. New designs pave the way to industrial adoption of the technology in these promising application domains. HSI in precision farming It has been estimated that agricultural use eventually will account for 80 percent of the commercial drone market. Farmers may use drones to tailor use of pesticides, herbicides, fertilizers and other tools based on necessity at specific points in a field – a process known as precision farming. If managed with precision, crops such as pears, potatoes, sugar beets and others can yield a more profitable harvest. Hyperspectral sensors can help farmers monitor nutrient levels, water availability, ripeness and disease conditions based on automated hyperspectral fingerprinting, rather than visual appearance. For the farmers, this will result in more consistent crops and better harvest timing. It will also save money they would otherwise spend overusing resources. These methods will also ease pressure on the environment by reducing nutrient and other runoff into nearby rivers and streams. Until recently, hyperspectral systems were too heavy, expensive and complex to add to the payload of drones. The cost of today’s agricultural drones may be as low as a few thousand dollars for a weight of a few hundred grams. This means that each piece of additional payload should cost only a fraction of that and be very lightweight. With the advent of these new high-quality cameras, each weighing less than 30 g, hyperspectral imaging is poised to support an increasingly data-driven agriculture. Meet the author Jan Provoost is science editor at imec in Leuven, Belgium; email: jan.provoost@imec.be.