The business of surveillance, whether by still or video camera, is booming. It’s being driven by military, industrial, and commercial customers as well as consumers. According to a report by BIS Research1, the video portion of the market will grow at over 16% per year, approaching $110 billion by the end of 2023. In addition to meeting this growing demand, vendors must satisfy the need to capture high-quality images in all types of lighting. As a result, they are increasing resolution and improving performance while cutting costs. The flood of surveillance data also presents challenges that may be solved at the edge of the network through machine learning. Since surveillance systems must “see” in all kinds of conditions, they use both visible and IR imaging technology. ON Semiconductor of Phoenix makes image sensors in the visible range that are used in a variety of settings, according to Jason Whetstone, senior product marketing manager for the company’s Intelligent Sensor Group. He said 24-hour surveillance monitoring requires the company’s chips to produce high image quality in both bright daylight and in situations with virtually no light. “ON Semiconductor has worked on specific imaging technologies to enable better low-light and no-light imaging,” Whetstone said. “Advanced pixel techniques have brought forward sensors that can image in color down to 0.1 lx.” For comparison, this is less than the amount of light of a full moon on a clear night. But even this level of imaging quality is not adequate for all situations. So the sensor maker has implemented enhanced processing techniques in its manufacturing line to improve the sensitivity of its silicon-based sensors in the NIR spectrum, which runs from 750 to 1000 nm — just above the visible. This enhancement consequently enables systems to use NIR illumination, allowing more detailed imaging in the dark, without having to use a light source that is visible to the eye. Details of buildings that are hidden by fog from visible imaging (top) are revealed by SWIR imaging (bottom). Courtesy of Xenics NV. There is a downside to the drive toward achieving quality imaging in low-light conditions. In bright sunlight, illumination intensity can exceed 100,000 lx. Surveillance systems must be able to handle this photon onslaught while permitting the clear imaging of objects in sunlight and shadows. The solution is to have a high dynamic range, enabling imaging under widely varying levels of illumination. ON Semiconductor has developed several techniques to achieve this range, including a multiple-exposure method that captures and then combines several different narrow dynamic-range exposures of the same subject. Using this technique, devices can image over a dynamic range of 120 dB, enough to accommodate objects in shadow on a bright day, according to Whetstone. While the technique is used in other vision and imaging applications, Whetstone said surveillance is seeing a “resolution race.” Full HD video is no longer sufficient, with some systems now opting for 4K resolution, roughly a four-fold increase in the number of pixels. “Many times, these higher resolutions are being used for object tracking, digital zooming, and region-of-interest marking,” he said. This move to higher resolution has a ripple effect far beyond the surveillance camera itself. The data generated grows in step with the increase in resolution, and this image data must be moved through a network, stored, and then analyzed. The entire backend infrastructure of a solution must be able to handle the amount of data produced by a 4K sensor. The transition to this higher resolution in mainstream applications is just beginning and will take several years to complete, according to Whetstone. Another emerging trend involves reducing power consumption, he said. More cameras are being deployed, and many of these new systems are battery powered. Thus, there is a demand to cut power consumption so solutions can run as long as possible on a single charge. One solution is to make the camera smarter. In a surveillance application, usually nothing needs to be imaged. Thus, most of the time the camera can sleep, Whetstone said, and the entire system only awakens when it detects movement or some other trigger. Some of these requirements and trends work against one another. The need for high resolution conflicts with the need for long battery life. For this reason, Whetstone said there is no single, best solution. Instead, trade-offs must be made that ultimately result in the most beneficial combination of features for a given product or application. The need to select from a range of possible solutions extends to the basic imaging technology used in surveillance. For instance, visible and NIR imaging works in sunlight and even in full moonlight, but in rain, fog, or on very dark nights, IR imaging may be the best or possibly only answer. Shortwave-IR, or SWIR, sits above the NIR, running roughly between 1- and 3-µm wavelengths. Sensors that can image at these wavelengths can work in poor weather conditions such as haze or fog. In these conditions, said Gary Chung, marketing officer for Xenics NV of Leuven, Belgium, SWIR imaging is superior to other parts of the spectrum. Thermal IR cameras — such as this mobile system mounted on a truck — can detect objects that are warmer than the surroundings. The cameras thereby enable surveillance at night. Courtesy of FLIR. Xenics makes IR cameras and sensors, and is active mainly in the SWIR arena. The detectors of such sensors are not made of silicon; they are instead made of indium gallium arsenide, indium antimonide, or a different material. For surveillance applications, SWIR enables unique capabilities. “SWIR hyperspectral imaging, for example, is used to identify and classify objects of interest by looking at their spectral signatures,” Chung said. He predicts continued improvements in the resolution of IR cameras and sensors. These advancements are closing the resolution gap between visible and IR systems, thereby allowing IR technology to find homes in more applications. Thermal IR imaging, running from a 3- to ~15-µm wavelength, detects the heat emitted by objects and provides the ability to extract additional information that can be useful in surveillance. For instance, thermal imaging can determine whether the engine in a parked car is hot. This, in turn, can help establish how long a vehicle has been stationary. Many implementations of IR sensor technology have required cooling of core components to cryogenic temperatures, thereby increasing power consumption and creating a problem for battery-powered applications. Wilsonville, Ore.-based FLIR Systems has developed a technology that allows a higher operating temperature (HOT) for the imaging core, according to Dan Walker, vice president of product management. The company’s new offering based on this HOT technology captures images in the MIR range, between 3 and 5 µm. It can provide high-definition or better resolution with small pixels, thereby allowing smaller optics for the same range and field of view as older systems, Walker said. “HOT FPA (focal plane array) technology allows camera providers to increase the number of pixels, shrink the camera, reduce power, and reduce cost,” he said. “So, it’s a virtuous cycle.” Sensors for surveillance applications, such as this one to the left of the dime, pack megapixels into small areas. They are designed and engineered for illumination levels from full sun to bright moonlight. Courtesy of ON Semiconductor. Finally, no matter the wavelength employed for surveillance, as imaging resolution increases, there is a growing challenge in handling and interpreting the information gathered. For those tasks, the advent of new interface standards, along with machine learning and artificial intelligence (AI), may offer solutions, according to Nicolas Baroan, business development manager at New Imaging Technologies of Paris. The company makes both visible and SWIR imaging systems. No matter the wavelength employed for surveillance, as imaging resolution increases, there is a growing challenge in handling and interpreting the information gathered. New interface standards, along with AI, may offer the solution. Faster interfaces make it possible to acquire data from a camera more rapidly. This faster speed consequently makes it more feasible to initiate a real- time response, such as sounding an alarm, alerting security personnel, or shutting a door or gate. Ultimately, machine learning and AI may spell the end of a long-time surveillance and security staple — the need for a human operator to stare at a bank of screens that display a video feed received from multiple cameras. The new technology of AI could have a profound effect on this picture. This is particularly true if machine learning technology can be pushed to the edge of the surveillance system and perhaps into the cameras that sit at the end of the system. This will reduce the amount of data flowing through the network and into storage. It will also mean the work of interpreting video and image data may no longer be such a labor-intensive undertaking. “We will probably move from human-based surveillance to machine-based surveillance,” Baroan said. “There will no longer be people watching a screen for 24 out of 24 hours, but rather an ‘intelligent camera’ delivering useful information to the security manager.” Reference 1. BIS Research (2017). Global video surveillance market — analysis and forecast, 2018-2023, www.bisresearch.com/industry-report/global-video-surveillance-market-2023.html.