From Crop Science to Space Exploration, Optical Sensing on the Rise

MARIE FREEBODY, CONTRIBUTING EDITOR, marie.freebody@photonics.com

From smartphones, smart homes and autonomous vehicles to crop science, food inspection and space observations, optical sensors are finding increasing use in the commercial sector.

Image sensing evolved from the military arena to commercial applications such as remote sensing, advanced machine vision, the medical/biotechnology world, and even into artwork and antiquities. Once size, weight and cost were driven down, there was an immediate upturn in the market; and this growth has been exponential.

IRIS is a small satellite with a powerful payload that takes high-resolution images of the sun’s interface region — the birthplace of solar storms. Courtesy of Lockheed Martin.

“There aren’t too many takers for a $20 million hyperspectral sensor, which does exist, but at less than five percent of that with performance attributes nearly equal, you start seeing interest from a range of commercial uses,” said Christopher Van Veen of hyperspectral imaging specialists Headwall Photonics in Bolton, Mass. “The multimillion-dollar hyperspectral instruments aboard defense aircraft and satellites represented a springboard for companies like Headwall to take the optical/mechanical technologies and refine them for commercial pursuits while driving cost out.”

At thermal imaging specialists Flir Systems Inc., based in Wilsonville, Ore., an increasing awareness of the technology has led to a vast number of new users in commercial industries, including for home repairs, heating and ventilation contractors, plumbers, and electricians, to name a few.

Headwall Photonics’ hyperspectral imaging sensors are used at one of the world’s largest almond growers and distributors, Travaille & Phippen in Manteca, Calif. The company inspects the almonds along conveyor lines using Micro-Hyperspec hyperspectral sensors paired with hyperspectral software and robotics. The sensor uses algorithms to discern ‘good’ from ‘bad’ with very high spectral and spatial resolution. The software communicates to robotic arms, which then know exactly where offending items are. Courtesy of Headwall Photonics.

“While the value of the technology was known by many professional users and those with military experience, lack of awareness, the acquisition cost, or both, prevented the vast majority of users from using it,” said Bill Terre, general manager and vice president of the OEM and Emerging segment at Flir.

In 2014, uncooled thermal imaging cameras came down from over $1,000 to $399 and many found the new value proposition made sense. Today prices continue to drop, with the latest models priced at less than $200. For many professionals, the more modest capital expenditure was what they needed to give it a go.

“Many have started with entry-level thermal imagers like the Flir ONE, but over time have migrated to more capable equipment. Even though this upgrade may come with a higher cost, we believe this trend will only increase over time,” said Terre.

Smartphones get smarter

Today’s cellphones take great pictures, but tomorrow they’ll assist doctors in diagnosing patient health. With sensor technology finally maturing to a mass-production level, tasks that were typically handled by people are increasingly being taken on by optical sensors.

With over 1.4 billion smartphones purchased in 2016, according to research firms International Data Corporation and Statista, a massive economy of scale has led to meaningful technology advances and cost reductions in many areas, including visible sensing and — on a lesser scale — thermal sensing.

“Thermal sensors have found application in mainstream smartphones via two avenues: aftermarket attachments like the Flir ONE and direct integration in smartphones such as the CAT S-60 handset,” Terre said. “These two vehicles account for hundreds of thousands of new users of thermal technology in 2016 alone. This significant increase in adoption has facilitated new awareness of the capabilities of thermal imaging. As a result, the industry has seen a strong increase in demand.”

Shrinking optics and silicon elements

One of the most daunting challenges Flir faced was conforming to the total volume requirement in smartphones of less than 250 mm2. The development of wafer scale vacuum packaging for the focal plane arrays (FPAs) was necessary to meet cost and Z-height maximums.

“Optics, constructed using only silicon elements and manufactured purely at the wafer level, were necessary to address both cost and volume limitations,” Terre said.

An image resolution significantly less than quarter video graphics array 320 × 240 became fundamental in tackling the unprecedented cost constraints. Unfortunately, the image appearance of an 80 × 60 or even a 160 × 120 FPA is far too little resolution for consumers accustomed to viewing the abundance of high-definition imagery seen today.

Ultimately, very sophisticated signal processing algorithms were necessary to increase the perceived image appearance, which is carried out by the general processing unit and image signal processor in modern handsets.

Flir also combined a thermal image with the high spatial frequency content of the visible image, known as multispectral imaging, or MSX, to increase the apparent resolution beyond that of the native focal plane array size.

Multispectral color-tunable source for wide angle lens commercial camera calibration. Courtesy of Labsphere.

“All of the advancements required for compatibility with the high-volume handset market have been realized with Flir’s Lepton family,” Terre said. “With two resolutions and pixel pitches — 80 × 60-17 μm and 160 × 120-12 μm — the Lepton sensors have been directly integrated into mobile handsets, mobile accessories, self-contained breathing apparatus, micro UAVs, personal vision systems and test/measurement equipment, to cite just a few examples.”

Smart home

Sensing applications using cameras and IR sensors, mostly on battery operated devices, are a critical component to many of the products in the smart home market.

Wolfgang Schmidt, senior product manager of IR Sensing at Excelitas Technologies in Waltham, Mass., expects this trend to continue as the market grows — driving size and power requirements down, and calling for enhanced performance and function.

Excelitas Technologies launched its 1 × 4 Pulsed Laser Diode Array for lidar applications at Photonics West 2017. The company believes the future will see more and more intelligence being integrated into the sensor module. Courtesy of Excelitas Technologies.

“Some of our latest breakthroughs focus on digital sensors. After the first digital sensor was introduced in 2010, we provided a low-power digital infrared sensor with only 5 µA /1.8 V power consumption in 2016,” said Schmidt. “Additionally, our new CaliPile Multi-Function Infrared Sensor represents a bridge between motion and temperature measurement applications.”

CaliPile intelligent IR sensors combine motion sensing, presence detection and temperature measurement in a single digital thermal infrared package. It features selectable frequency filters and levels that allow users to set the product into one of the three major operation modes.

Crop and food inspection

Remote sensing is perhaps the biggest market for optical sensors, with applications ranging from crop science to greenhouse gas emissions and plant phenotyping to geological analysis.

In crop science you can “see” the telltale signs of crop diseases, which left unchecked and unseen can lead to entire economies collapsing. For example, the value of being able to detect disease very early in coffee beans in South America, in citrus fruit in Florida or wine grapes in Napa is incalculable.

“By making sensors smaller and more affordable, two things happen: more scientists can afford them, and more UAVs [unmanned aerial vehicles] can carry them,” said Headwall’s Van Veen. “And we find at Headwall that the remote sensing platform of choice is the UAV. They are nimble, tactical, easily deployed and affordable. It’s important that the payload be matched to the craft, which is why size and weight are both key metrics. Too heavy or too big, the UAV’s capable of lifting them are fewer.”

The advanced machine vision market is also seeing the benefits of hyperspectral imaging. This industry has lived with rudimentary RGB camera systems for a long time, but governmental oversight within the food inspection industry is causing many leaders to look at new sensing technology that gives a crisper view of their product under inspection.

Headwall’s sensors are today in place in central California at a leading almond producer. There, the hyperspectral sensors sit, like silent sentinels above the high-speed production line, examining every single almond and determining whether it passes the grade. Finely crafted algorithms are used to grade the nuts and once the sensor detects an out-of-spec nut or berry, software immediately communicates its spatial position on the line to downstream robotics for ejection.

“The Headwall sensors inspecting almonds all use a diffractive approach based on gratings and mirrors. The diffraction gratings are all ‘originals’ rather than replicates,” said Van Veen. “That means each instrument is an exact duplicate of its neighbor, which is crucial when multiple inspection lines are all looking at the same thing. Instrument repeatability and consistency matter from one sensor to the next.”

Lidar

The automotive lidar market is exploding and pushing the boundaries of what is and will be available on the sensor front, according to experts. Thales is involved in active optical sensing, mainly for lidar and active imaging systems, where technology advances are helping to expand the market.

The lidar market is exploding, with technology advances helping to expand it. Courtesy of ecliptique Laurent Thion.

“There is a diffusion of coherent lidar technology for both defense- and civilian-oriented purposes. For active imaging, main progresses come from additional contrasts provided by polarimetric imaging and multispectral imaging,” said Daniel Dofli, director of research for physics applications at Thales Group of France. “Fiber-based technologies developed originally for the telecom markets are now largely used in lidar systems, with a clear impact on cost, size and, [to] some extent, on performance.”

Dofli predicts that the emergence of photonic integrated circuits will trigger a second revolution in terms of increased compactness and reliability on one hand, and of tremendous cost reduction on the other.

Excelitas Technologies launched its latest 1 × 4 pulsed laser diode array for lidar applications at Photonics West 2017. It combines high-efficiency multi-cavity laser chip technology with a small form-factor surface mount device.

Its 1 × 4 linear configuration minimizes space requirements in the assembly to enable smaller, less costly optical components for applications requiring high reliability such as autonomous vehicles and drones.

“Innovative products like these are just the beginning. In the future, we will see more and more intelligence being integrated into the sensor module,” Schmidt said.

Low-uncertainty uniform source for testing European Earth-observation satellites. Courtesy of Labsphere.

Software must keep up with data

Software not only manages the sensor, for example, instructing it when to collect data based on geographical coordinates, but also handles the key post-processing tasks once you’ve collected several gigabytes of hyperspectral data. What do you do with all that data? How do you go from raw image data to actionable solutions and answers to your questions? That aspect of ‘software’ is every bit as important as anything else.

When it comes to handling the huge amounts of data from optical imaging, there can be fewer disciplines that generate more than satellite-based sensors for earth observation. In order to enable climate change prediction and a better understanding of the complex chaotic system driving these phenomena, data on the order of greater than two percent uncertainty needs to be gathered for 25 years on orbit.

“The sensing technology will only go as far as the data will allow it to go,” said Chris Durell, director of Marketing, Imaging Products, at Labsphere in New Hampshire. “We can now take a massive amount of information, but if we cannot extract the needed data to make decisions, this is just wasted information. Better data extraction is needed to enable the next level of growth in optical sensing.”

Sensing Spacecraft

Global security company Lockheed Martin has a long history not only in the defense and security arena, but also in space sciences and specifically NASA programs. In its latest mission into space, Lockheed Martin is helping NASA begin the hunt for dark energy, a mysterious force powering the universe’s accelerating expansion.

The company is comleting a study to build an important part of the primary optical instrument aboard the Wide Field Infrared Survey Telescope (WFIRST), whose mission aims to uncover hundreds of millions more galaxies and reveal the physics that shapes them.

The Near Infrared Camera (NIRCam) is the primary optical instrument aboard NASA’s next deep space observer, the James Webb Space Telescope. Courtesy of Lockheed Martin.

“From solar observing satellites to NIRCam — the near-infrared camera for the James Webb Space Telescope — Lockheed Martin has been at the forefront of space sensing,” said Jeff Vanden Beukel, Lockheed Martin program director for the Wide-Field Optical-Mechanical Assembly (WOMA). “Our Advanced Technology Center is focused on developing next-gen and gen-after-next technologies. We see customers looking to explore larger telescopes and longer wavelengths, as well as ways to reduce payload size and cost for future missions.”

Innovative telescope and back-end relay optical designs have allowed for more compact systems with wider fields of view. Imaging focal plane arrays have also continued to progress with increased pixel count and improved sensitivity.

“Spacecraft isolation, alignment integration and test, sensor technologies, telescope design — these are all areas that will help shape the next generation of sensing spacecraft,” said Vanden Beukel.