High dynamic range (HDR) technology has altered image capture by enhancing clarity and preserving details in both bright and dark regions, an advancement that is proving to be instrumental in sectors such as advanced driver-assistance systems (ADAS), autonomous mobile robots, and autonomous vehicles. A close-up view of interspersed large and small pixels. In low-light conditions, the large pixel’s heightened light absorption capabilities come to the forefront, while in brighter conditions, the smaller pixel’s extended exposure time prevails. Courtesy of e-con Systems. Dynamic range in embedded vision systems enables the camera to capture a wide range of luminance, spanning from the darkest to the brightest parts of an image. In environments with varying light levels or scenes featuring extreme brightness and darkness, embedded vision systems equipped with HDR play a pivotal role. Advantages of HDR in embedded vision systems include: • Image quality: A heightened dynamic range empowers the camera to capture intricate details in shadows and highlights, ensuring the production of images in which details remain visible and clear even in conditions with stark lighting contrasts, such as direct sunlight and shade. • Improved processing capabilities: The amount of information captured by a better dynamic range enables advanced processing algorithms to be applied. This facilitates enhanced object-background distinction and the analysis of scenes with intricate lighting conditions, contributing to more accurate and efficient processing. • Reduced artifacts and noise: HDR imaging mitigates the appearance of noise and artifacts that commonly plague images with extreme contrasts. By capturing more detail in light and dark areas, the reliance on aggressive image processing, which can introduce noise, is minimized. • Accurate color representation: In scenes with a heightened dynamic range, limitations in a camera’s dynamic range can result in washed-out or unnaturally saturated colors. HDR ensures that colors are accurately represented, offering a more reliable depiction of the scene. Traditional multiexposure HDR Traditional multiexposure HDR has long been the go-to technique for capturing the full spectrum of luminance in a scene. Operating on the fundamental principle that different exposure levels suit various parts of a composition, this method combines short, medium, and long exposures to unveil details across highlights, mid-tones, and shadows. While this approach has proved to be effective, its inherent limitations merit consideration. To achieve a subpixel high dynamic range (HDR) image, four subpixel images are combined and processed by the image signal processor. Courtesy of e-con Systems. Traditional high dynamic range (HDR) images of moving scenes often show motion artifacts. Courtesy of e-con Systems. Motion artifacts In scenes featuring fast-moving subjects, the temporal gap between successive exposures can introduce undesired effects, such as ghosting or blurring in the final HDR image. Addressing this challenge is crucial for maintaining image clarity and precision, particularly in dynamic environments. Computational intensity The process of aligning and merging multiple exposures demands substantial computational resources. This computational intensity can pose constraints, especially in real-time applications or scenarios in which power efficiency is paramount. Striking a balance between computational demands and real-time processing remains a key consideration in optimizing HDR imaging. Camera hardware limitations The successful implementation of traditional multiexposure HDR hinges on the camera system’s ability to rapidly capture images with variable exposure settings. However, not all embedded vision systems possess the necessary hardware capabilities, which introduces a potential barrier to widespread adoption. Subpixel technology In the dynamic landscape of embedded vision, the emergence of subpixel HDR technology allows camera systems to capture a broader spectrum of brightness levels and deliver images that are not only vibrant but also true-to-life. This transformative technology, also known as subpixel HDR, has become increasingly popular due to its unique approach in enhancing image quality and color accuracy. Subpixel HDR technology operates by dividing each pixel on the sensor into two subpixels (SP1: large pixel, SP2: small pixel), dedicated to capturing the brighter and darker areas of a scene. Notably, the camera system captures images simultaneously rather than sequentially, leading to a reduction in motion blur and the delivery of more vivid and true-to-life images. Essentially, it ensures a more precise representation of colors, resulting in images that accurately mirror the natural color tones of the scene. Each individual pixel in this innovative design incorporates both a large and a small on-chip microlens (OCL). Specifically, the OCL of SP2 is strategically positioned within the gap section of the OCL of SP1. This unique configuration enables each pixel to feature a dual nature: one with “high sensitivity,” designed to illuminate dark subjects brightly, and the other engineered to resist saturation, even when exposed to intense light. The synergy of signals from these two distinct pixels results in a remarkable outcome — a signal-boosting and expansive dynamic range. This technological feat ensures that scenes with high contrast, such as backlit conditions or tunnel entrances and exits during daylight, experience no loss of detail in dark areas and no overexposure in bright regions. To achieve this dynamic range, a deliberate choice is made in allocating the subpixels. A larger subpixel takes on the responsibility for darker portions of an image, harnessing its capacity to collect more light. This greater light absorption, however, leads to saturation in brighter conditions. On the other hand, a smaller subpixel is dedicated to mid-to-high light portions of the image, because it can be exposed for longer without easily reaching saturation due to its lower light collection efficiency. This interplay between the large and small pixels serves a dual purpose. In low-light conditions, the large pixel’s heightened light absorption capabilities come to the forefront, while in brighter conditions, the smaller pixel’s extended exposure time prevails. The net result is a seamlessly extended dynamic range, ensuring optimal performance across a spectrum of lighting scenarios. During each frame exposure, the four lines representing SP1_HCG, SP1_LCG, SP2_HCG, and SP2_LCG are combined, resulting in a 24-bit output value. This combined output is then processed by the image signal processor (ISP), an integral component of the camera module. The ISP transforms the data into an 8-bit YUV format, widely compatible with most applications. The essence of this subpixel architecture lies in its ability to capture an impressive HDR of 120 dB. This is achieved through the meticulous merging technique described above, showcasing the technological prowess of the camera module in delivering enhanced image quality across a wide spectrum of lighting conditions. Range of benefits The use of subpixel HDR technology presents a range of benefits, particularly in scenarios involving dynamic scenes with moving elements. In contrast to conventional HDR methods that rely on varying exposures that could lead to motion artifacts or blur, the subpixel HDR technique offers distinct advantages. One notable improvement is its ability to circumvent the challenges associated with motion blur. Unlike traditional approaches, subpixel HDR captures images simultaneously, eliminating the potential for motion artifacts. This ensures that moving objects within the scene are depicted with clarity and precision, appearing sharp and well defined. The result is a set of images featuring moving objects with enhanced sharpness and intricate detailing, setting it apart from traditional HDR techniques. This transformative technology not only ensures superior HDR performance but also addresses challenges such as motion blur, making it an indispensable asset in applications ranging from ADAS in automotive settings to surveillance and industrial vision systems. Empowering automotive vision In the automotive realm, subpixel HDR technology is reshaping the landscape by providing the precise perception and interpretation of complex road scenarios, even under unpredictable lighting conditions. This technology boasts exceptional HDR performance, reaching up to 120 dB (high HDR), thereby enabling ADAS to capture detailed information in diverse lighting environ- ments without succumbing to motion blur. For autonomous vehicles, where environmental interpretation is para- mount, subpixel HDR ensures consistent and detailed visual information. By assigning different exposure values to individual subpixels within a pixel array, this technology excels in producing balanced exposures with detailed highlights and shadows, seamlessly transitioning between extreme lighting conditions. Simultaneous high dynamic range (HDR) images of moving objects. By assigning different exposure values to individual subpixels within a pixel array, the technology can produce balanced exposures, even of objects in motion. Courtesy of e-con Systems. In the realm of autonomous vehicles, subpixel HDR directly enhances performance by identifying obstacles, pedestrians, and traffic signs across a wide array of lighting scenarios. Notably, this is achieved without the motion artifacts associated with traditional multiframe HDR techniques, because subpixel exposures are captured simultaneously. Surveillance Subpixel HDR technology plays a pivotal role in surveillance applications, offering the ability to capture clear images in diverse lighting conditions. This is essential for accurate identification and efficient activity monitoring. In high-contrast environments, surveillance cameras equipped with subpixel HDR can capture intricate details in both bright and dark areas, enabling the precise identification of individuals or objects. The industrial landscape benefits significantly from subpixel HDR, partic- ularly in quality control processes. Vision systems equipped with this technology excel at detecting defects and irregularities on assembly lines with unparalleled precision. The capability to handle complex lighting scenarios, from intense glares to shadowed areas, ensures that cameras maintain production standards. This level of precision is indispensable for automated systems, allowing them to reliably detect and address potential issues in real time. Challenges in subpixel HDR • Cost and complexity: The adoption of subpixel HDR technology necessitates intricate sensor designs and advanced image processing algorithms, leading to an increase in the overall cost and power consumption of the camera system. Consequently, this renders it less suitable for specific consumer-grade applications in which cost-effectiveness and power efficiency are paramount. • Poor image quality: Diminished image quality arises due to increased noise levels in elevated automotive temperatures. • Optics selection: Selecting a lens for subpixel image sensors can be challenging because it is difficult to match the lens resolution to the sensor resolution. Subpixel HDR technology is reshaping how images in automotive and surveillance applications are perceived and processed. Its ability to capture a broader range of brightness levels, coupled with reduced motion blur, positions it as a transformative force in the world of embedded vision. In navigating the possibilities and challenges of embedded vision, subpixel HDR continues to illuminate the path toward enhanced visual experiences across diverse industries. Meet the author Prabu Kumar is the chief technology officer and head of camera products at e-con Systems. Kumar has more than 15 years of experience in the embedded vision space, possessing a deep knowledge of USB cameras, embedded vision cameras, vision algorithms, and FPGAs. He has built 50+ camera solutions spanning various domains, such as medical, industrial, agriculture, retail, biometrics, and more. Kumar’s current focus is to build smart camera solutions that power new-age AI-based applications; email: prabukumar@e-consystems.com.