Getting the Most Out of Your Image

Sarah Sookman is a media relations specialist for the marketing and communications department at Matrox Imaging in Dorval, Quebec, Canada.

For developers of scientific or industrial image-processing applications, robustly and accurately extracting the vital information contained in each pixel presents a constant challenge. Recent developments in camera, frame grabber and algorithm technologies have contributed to raising the standards for picture quality, data rates, processing capabilities, robustness and accuracy.

The front of the line

At the front end of an application is the camera (or sensor). Typical analog cameras provide 640 x 480-pixel, 8-bit resolution at 25 or 30 fps. Many imaging applications, though, demand higher resolution and frame rates. As digital cameras made their appearance, those in industry began to see their benefits: Converting analog data to digital was no longer necessary, and images at a greater resolution and bit depth could be acquired. With digital cameras, image data can be transmitted directly to the PC, eliminating the need to convert the digital data to analog and back again. The benefits of an all-digital interface spawned research efforts in this direction.

A few years ago, we saw the introduction of IEEE-1394 (FireWire), which was both a standardization and a simplification intended for consumer electronics, including digital cameras. Low-voltage differential signal and RS-422 are fairly common digital interfaces but are insufficient for some applications, typically those requiring high speed and/or high frame rates, such as web and semiconductor inspection. In 2001, Camera Link made its debut -- a digital interface standard based on Channel Link technology from National Semiconductor that answers this demand for high speed and resolution (Figure 1).

Figure 1. Increased bandwidth through a smaller interface is the trend for camera-to-PC connections. Pulnix's TM-6710 camera (left) has a traditional 8-bit low-voltage differential signal connector, but the company's TMC-1000 outputs 24 bits of data and four control signals through a smaller National Semiconductor Channel Link interface.

Many camera and frame grabber manufacturers have contributed to the Camera Link standard's quickly becoming the interface of choice for systems integrators. Channel Link uses a variation on the low-voltage differential signal that is already used for transmitting parallel data and that traditionally requires 56 conductors to transfer 28 bits of data. The number of control bits is not standardized and is highly dependent on the camera, but it would typically include five signals for 10 wires. The signal's performance is difficult to measure as well, but data transfer rates of 160 MB/s are typical.

Several versions of Camera Link exist. The version used for industrial cameras requires only 11 conductors to transfer 24 bits of data, and four control bits. The reduction in the number of conductors gives the technology the edge on data rates, transmitting an impressive 680 MB/s (Figure 2). Other benefits include standard connectors and cable lengths up to 10 m. Furthermore, Channel Link technology not only eliminates the need for a software protocol like IEEE-1394, simplifying camera design, but also removes the requirement for complex communications software.

Figure 2. Channel Link interface technology (top) -- essentially a "hybrid" of low-voltage differential signal and digital serial interfaces -- serializes parallel data to reduce wire numbers, size and shielding materials.

The IEEE-1394 standard was established in 1995 as a digital serial interface and interconnect for consumer electronics and computer peripherals. Because it was intended for incorporation into a broad range of products, such as professional audio/video and storage, scanning and printing devices, a number of protocols have been designated. Hardware that supports IEEE-1394 industrial/scientific digital cameras follows the protocol outlined in the Instrumentation and Industrial Digital Camera specification.

A single six-wire cable can power the camera and carries both the video data to the PC and the communication signals from the PC to the camera. IEEE-1394a supports maximum data transfer rates of 400 Mb/s but is limited to a cable length of 4.5 m. With the emerging IEEE-1394b, cables can be made from other materials, such as plastic or glass fibers, and these cables support maximum lengths of 50 or 100 m, depending on the data rate and the material. In addition, IEEE-1394b supports maximum data rates of 800 Mb/s (3200 Mb/s with a glass fiber optic cable) and has the potential to go beyond those rates.

As camera technology continues to improve, today's industrial and scientific imaging applications require the processing of more and more pixel data. For some applications, especially those that employ the Camera Link interface, pixel processing rates on the order of hundreds of megabytes per second are the norm. As a result, manufacturers of frame grabbers have had to develop high-throughput products to satisfy the demand for and to avoid the loss of data.

The bus is overcrowded

The peripheral component interconnect (PCI) bus is the enabling technology that gave systems designers the means to stream image data into system memory in real time, where it could be processed by the ever-increasing power of host CPUs. This paved the way for the adoption of the PC as a real-time imaging platform.

In the beginning, the PCI was positioned as a bus that delivered 132-MB/s performance, but with the chip sets found on early PCs, the actual sustained performance was closer to 20 to 30 MB/s. This benchmark is a key factor for reliable image capture to host from a PCI bus frame grabber. With today's PCs, this is not a problem, and properly designed frame grabbers can deliver video image data to host memory at rates surprisingly close to the theoretical maximum throughput.

Another performance handicap was that some early PCI bus frame grabber designs were not PCI bus masters. This type of interface is necessary for a frame grabber to transfer data to system memory for processing, without the continuous and direct involvement of the CPU. If the host CPU is busy reading pixel data, it is not processing. But as time went on, designers of PCI chips figured out how to make full use of that bus design by implementing functionality such as bus mastering to optimize throughput and system performance.

For applications that use IEEE-1394 cameras, the PCI bus should be able to adequately transfer and process the data. To maximize the capabilities of the Camera Link specification, however, developers needed a faster, wider bus.

The PCI Special Interest Group developed a high-performance enhancement to the conventional specification, called PCI-X, to resolve the bandwidth issue. This technology, with its increased performance, addresses the demanding input/output requirements of a new generation of high-bandwidth applications like Gigabit Ethernet and high-speed Camera Link cameras. Version 1.0 of PCI-X specifies a 64-bit connection running at speeds of 66, 100 or 133 MHz, resulting in a peak bandwidth of 528, 800 or 1064 MB/s, respectively.

Image-processing algorithms in use today can be described as more robust and extremely fast. The theories behind a number of techniques, such as geometric pattern matching and feature extraction, have been with us for decades. Increased processing power has made investigation of the algorithms that were unusable for industrial applications a worthwhile task. But processing power is only part of the story. Merely plugging components together is not going to produce a usable algorithm. Developers must fine-tune those components so that they satisfy the performance objectives of speed, robustness and accuracy.

Robust algorithms

In a semiconductor wafer-alignment application, the wafers are polished several times during the production process. After polishing, the fiducials may be severely altered, so the algorithms must be robust enough to locate any that have been, and the varied or nonuniform lighting environments for this application must be taken into consideration. At the same time, these procedures cannot slow the production rate. CPUs are designed to speed up multiple calculations.

Five or six years ago, technologies such as Single Instruction Multiple Data, developed by academia; Altivec by Motorola; and MMX/SSE2 by Intel, pushed this further, enabling those calculations to be performed in parallel. Many image-processing operations, such as normalized gray-scale correlation (pattern matching) and blob analysis, can be performed on an image with a more "brute force" approach. These algorithms are less dependent on the actual image content, and the same instruction is performed on multiple data. Algorithms for geometric pattern matching and feature extraction use high-level heuristics and involve hundreds of decisions based on the analysis of individual pixels.

Because 1-GHz and faster processors can obtain results for complex algorithms within milliseconds instead of minutes, branching is no longer as serious a problem. As a result, complex algorithms are becoming commonplace in industrial applications. Many newer image-processing tools are so powerful that the requirements of what constitutes a good image have been relaxed somewhat.

In a traditional industrial setting, many factors can interfere with acquiring image data from a camera. Lighting has always been a problem. Shadows can fall in the camera's field of view, as can overhead reflections. Uniform lighting, therefore, is very difficult to accomplish. Blob-analysis operations are quite common in industrial or scientific applications and can be used to find simple shapes. These operations are binary and rely on consistent lighting. Furthermore, once the image is transferred to binary form, results can be obtained only with pixel accuracy. A feature-extraction tool, such as Matrox Imaging's Edge Finder, surpasses these capabilities by showing a tolerance to lighting variations and by using the relationship of a pixel to its neighbors to find line crests and contours.

Figure 3. Geometric pattern-matching algorithms are robust enough to find multiple models or occurrences of models, even if they are occluded. They can locate a model at any scale or orientation, even in environments with nonuniform or poor lighting.

Geometric pattern matching is another technique that is now possible because of increased processing power (Figure 3). This method uses defined features instead of actual pixels to locate a model in a target image. For example, if the algorithm is searching for a polygon with four corners, the corners and the relationship between them are used to find the shape. Because the specified distance, or the distance ratio, between these features is noted in the target model, geometric pattern matching can locate the object, no matter what the angle or scale.

Matrox's Geometric Model Finder tolerates significant occlusion. In a pick-and-place robotic operation, where a robot grabs a part out of a bin, the benefits of geometric pattern matching are compounded: It can search for multiple models and can locate models with significant occlusion in nonuniform lighting conditions.