With the U.S. and the U.K. deploying more troops overseas, and given the global increase in the frequency and lethality of terrorist attacks, it’s little surprise that versatile unmanned aerial vehicles (UAVs) are playing a bigger role than ever before to tackle a new era of complex military issues. A key trend is the availability of increasingly compact sensors, which is enabling imaging systems to take to the skies. The aerial imaging market is expected to garner $2.84 billion by 2022, registering a compound annual growth rate of 12.9 percent during the forecast period 2016-2022, according to a market report by Allied Market Research. Lockheed Martin’s HALE-D is an untethered, unmanned lighter-than-air vehicle that will operate above the jet stream in a geostationary position to deliver persistent station keeping for a variety of missions. Courtesy of Lockheed Martin. Challenges of weight, size and ability to withstand environmental damage must be balanced with image quality, coverage and efficient data transmission, as well as the ability to make sense of the huge amounts of information collected. Today, there are thousands of UAV platforms available, each serving a different purpose and often flying at distinct altitudes. From the sizeable high-altitude imaging drones such as the U.S. Department of Defense’s high-altitude, long-endurance (HALE) Global Hawk to the smallest FLIR Black Hornet nano-UAV, they provide intelligence, surveillance and sometimes target acquisition and reconnaissance for military personnel. UAV gesture recognition is now possible thanks to advanced optics and software. Courtesy of Lockheed Martin. The Black Hornet reflects an important trend. As the world’s smallest and lightest unmanned aerial system, weighing less than 1 oz — about the weight of three pieces of paper — its purpose is to be a personal asset to the modern soldier. This is a move away from UAVs of the past decade that have been a company or battalion asset. “The Black Hornet is revolutionary in many ways,” said Adam DeAngelis, vertical marketing director for FLIR surveillance at FLIR Systems Inc. in the U.S. “It uses a single main rotor, versus the more common quad-copter design. About the length of a pencil, it can fly up to 25 minutes in close quarters quietly, at up to 25 mph for distances up to a mile and is virtually undetectable from 15 feet away.” Smaller UAVs are becoming a personal asset to the modern soldier. The Black Hornet, for example, is the world’s smallest and lightest unmanned aerial system. Courtesy of FLIR Systems Inc. Considerations such as size, weight and power (SWaP) are critical for UAVs, particularly ones such as the Black Hornet. Every platform needs the most efficient sensor package for the size of the vehicle carrying it aloft. In addition, effective energy harvesting, combined with superefficient onboard processors and advanced encrypted radio communication for live streaming video, is extremely important. “While imaging sensors are getting smaller and smaller, and high-definition (HD) image processing into real-time, actionable intelligence becomes faster, we expect to see smaller, lighter and more robust imaging sensors combined with new efficient software for the booming UAV market,” DeAngelis said. A schematic of a concentric imaging sensor comprising a diffraction grating that disburses/diffracts incoming reflected light seen through the image slit (lens). Concentric mirrors and the grating work in tandem to present a high-resolution image to the focal plane array across hundreds of spectral bands. Courtesy of Headwall Photonics Inc. FLIR’s Black Hornet is available in “day” and “night” versions with different cameras and provides a live video feed to the operator on a chest-mounted display. “We see a clear trend in thermal imaging becoming an increasingly important part of the UAV sensor package,” DeAngelis said. “Combining thermal imaging with EO [electro-optical] or other sensor technologies, with trends moving toward superresolution imaging on smaller UAV platforms, we see a big growth potential in the small/micro/nano UAV segment.” Advances in imaging components Technology has progressed in most of the elements of an imaging system for a UAV — from the HD detectors available today to multispectral imaging, onboard processing, ground-to-air communications and vice versa. Image resolution has improved with the advent of HDTV cameras, which benefit from increases in the number of pixels in a sensor’s array. In particular, focal plane arrays have evolved from a 320 × 240-pixel format to 640 × 512 pixels and, now, HD array formats of 1280 × 720 pixels or 1920 × 1080 pixels and above. “The use of HD sensors has revolutionized the design of last-generation electronics in a way that the throughput of all image chain has been multiplied by a factor of approximately seven,” said the project officer in charge at the European Defence Agency (EDA). “This leads to the use of optical slip ring in the azimuth gimbals, compliant at least to the 1.5 Gbps rate, high bandwidth cabling, latest generation of HW/FW for the electronics boards capable to perform the image processing algorithms on HD video frames.” When it comes to communications between the payload and the ground station, which is attended by an operator, there are big challenges. More HD sensors cooperating on the same platform means serious problems of bandwidth can arise. “While data compression (i.e., H264) is a good solution,” the EDA source said, “in some cases when true real-time is needed, it can show some limits in terms of latency.” Today, it is quite common to integrate a processing unit that is able to understand the acquired image and automatically detect the target of interest, so that only the region surrounding the targets is cued and transmitted at full resolution while the rest of the image is discarded or transmitted at lower resolution. The challenges of airborne imaging Size and weight are the obvious caveats when it comes to UAV deployment: By necessity, the imaging sensors need to be smaller and lighter to fit more compact payload bays. An integrated airborne solution aboard a multirotor UAV. In this photo is a SWIR imaging sensor, a high-performance GPS/IMU (inertial measurement unit), and also a data storage hub that provides a synthesized stream of data from all connected instruments — in this case, the SWIR sensor and the GPS/IMU. Courtesy of Headwall Photonics Inc. But that’s not all. For airborne applications, it also is necessary to have rugged sensors with few or no moving parts that can withstand the rigors of flight. In fact, the simpler the optical design — using diffraction gratings to manage the dispersion of reflected light, for example — the more reliable the instrument will be. U.S.-based Headwall Photonics Inc., an expert in deploying airborne imaging sensors, specializes in hyperspectral imaging to bring added analytical power to the picture. “The purpose of hyperspectral imaging is to collect actionable spectral data that provides an exceptionally high degree of discrimination and classification across a very wide field of view,” said Christopher Van Veen, marketing manager at Headwall Photonics. “In defense applications, this often comes down to the ability of a sensor to differentiate between materials that look similar to the human eye but exhibit unique spectral signatures.” Operating at wavelengths beyond the visible (into the infrared ranges), a hyperspectral imaging sensor collects a full spectrum for every pixel it sees. Proper classification of chemicals and materials comes down to resolution (both spatial and spectral), having a wide field of view that allows the UAV to fly a more efficient flight swath, and excellent signal-to-noise characteristics so the sensor can do its job even in less-than-optimal solar illumination conditions. When it comes to the optics, the ability of a grating to diffract a spatial line of the spectral components is fundamental. The “pushbroom” (line-scan) sensor design comprises a simple but elegant layout of a diffraction grating and concentric mirrors that present the focal plane array (FPA) with an accurate representation of all spectra for a line in the scene, on a frame-by-frame basis as the UAV flies. The combination of these frames is a hyperspectral data cube, which can be gigabytes in size. The problem is, as the UAV flies, it is subject to roll, pitch and yaw, which lead to unwanted effects to the image data. In some cases, a stabilizing gimbal will help, but often this needs to be combined with a high-quality GPS/IMU (inertial measurement unit), as well as orthorectification of the data during post-processing. The accuracy of spectral data is a function of hardware and software, which effectively manages and interprets the data so rapid decision-making can occur. “Software geared specifically to airborne applications is crucial, not only for orthorectification but also to manage the operation of the sensor while aloft,” Van Veen said. “Headwall’s ‘polygon tool’ allows users to input GPS coordinates within which the sensor will operate. Outside of those defined coordinates, the sensor will not collect data. This is very useful since data efficiency is fundamental to the effort.” Processing for navigation GPS information can be integrated with the sensors to allow for geolocation tagging of images, meaning one can reference the image on a map. According to the EDA, GPS systems also can be used to help sensors continuously track targets in uniform motion. “All of this imagery can be transferred via data link to operators with access to displays available at ground stations, inside ships, at air operations centers and distributed ground stations, and on tactical handheld systems like the Remotely Operated Video Enhanced Receiver (ROVER) and the One System Remote Video Terminal (OSRVT),” the EDA source said. With an endurance of approximately 45 minutes, the Indago drone can achieve ranges of up to 5 km with a handheld controller, and the range can extend beyond 10 km with directional communications devices. Courtesy of Lockheed Martin. As imaging technologies continue to evolve to meet the operational needs for militaries around the world, more and more effort is being placed on immediate or real-time processing and application of imaging products for navigation and other use. With that comes a need for more immediate understanding of the imagery and what that imagery means in the context of its environment. “As with ground-based systems, a big challenge is always how to get more information from the sensor to the user (human or machine) as quickly as possible,” said Joe Suhr, C4ISR Engineering Fellow at global defense company Lockheed Martin. “This requires more efficient and smart bandwidth. Specific attention also needs to be paid to SATCOM [satellite communications] for command and control of the sensor, as well as full autonomy.” At Lockheed Martin, engineers are enhancing real-time video from electro-optic and IR sources, including stabilization, noise reduction, haze reduction and contrast enhancement in both CPU and GPU environments. “We are also developing change detection services for still imagery and full motion video that include 3D world generation and colorized product generation,” Suhr said. “And with these high-fidelity visual products, information from other sources (such as radio frequency signals) can be fused into the picture to provide a multisensor view of the world that will continue to improve a UAV’s ability to react intelligently without human intervention.” As the amount of imagery data increases, analytics and technologies must keep pace to find and size the proper transport mechanisms and algorithms to sift through this data. This will help both human imagery analysts and artificial learning algorithms find that needle in a haystack. “This will continue to put pressure on the communication systems and data links to increase bandwidth and performance, while trade-offs will have to be made to determine if and how much processing should be done in the air to maximize the datalinks available,” Suhr said. “New compression strategies and prioritization schemes have to be considered in this trade space.” Expect more eyes in the sky Traditionally, the use of aerial imaging solutions was limited to defense, cartography and civil engineering. However, increased awareness regarding the benefits offered, as well as leaps in precision and capability, means that airborne imaging solutions have extended their applications to surveillance, intelligent inspection, disaster preparedness and risk study, and tourism, among others. Aerial imaging picks up storm damage on properties for rapid assessment of insurance claims after recent hurricanes in the southern U.S. Courtesy of EagleView Technologies Inc. For example, after recent hurricanes in the southern U.S., aerial imaging specialist EagleView Technologies Inc. has been sending drones into the fields for a client to process insurance claims quickly and from a remote location, saving the claims adjuster time and the insurance carriers money by giving them the information they need through high-resolution drone imagery. “With our drone technology, we’re able to use machine learning capabilities in conjunction with our drone imagery to identify storm-related damage on property structures through artificial intelligence, minimizing time to analyze images and increasing efficiency,” said Robert Locke, senior vice president of Government Solutions at EagleView. Escalating demand across various industries has been noted by U.K.-based market research firm Credence Research, which cites budding camera technologies including Hawk-Eye, Eagle-Eye and UltraCam Osprey as providing the necessary impetus for widespread adoption of aerial imagery systems. “With the rise in unmanned aerial vehicles, there has been growing interest in location-based services, especially for applications including remote monitoring, navigation and commercial advertising through aerial imaging technology across multiple verticals,” said Laveena Dsouza, business manager at Credence Research. Crop science is also proving to be fertile ground for imaging sensors mounted on UAVs. They are much more affordable and tactically efficient than manned aircraft or satellites and can cover smaller geographical areas with high resolution. This allows researchers interested in spotting disease conditions to fly over their fields and interpret data on a weekly or even daily basis. But with increasing surveillance applications comes a growing concern of intrusion into privacy, as well as the technology being used in nefarious ways. “There have been mixed views surrounding the use of drones,” Dsouza said. “While people are in approval of law enforcement using drones to track down fleeing suspects, they are wary about government agencies using drones to spy on innocent citizens. In addition, growing fear of terrorism among people has only added to the distrust, as the technology is thought to be used for delivering explosives or to scout out targets.”