For better or worse, security is increasing, and that means new opportunities — and challenges — for machine vision. Cameras and systems developed originally for inspecting widgets now are being pressed into service for surveillance and for gathering biometrics.Fingerprint reading at a distance, facial recognition in a crowd and detection of people who are lingering in a hallway all are security applications that make use of vision technology. A look reveals what is happening now and what could occur in the future with regard to surveillance and biometrics.It is important to get the full surveillance picture without causing data overload. On the left is a full-resolution, full-field-of-view shot of a highway. This 4-megapixel image at 15 fps at an 8-bit gray scale is more than 60 MB/s — too much for most wireless connections or a multicamera wired network. In the middle is the same shot, with every fourth pixel and fourth row. The road and vehicles still can be seen, but the bandwidth is less than 8 Mb/s. On the right is a digitally zoomed image, with all pixels present in a 4× smaller field of view. Again, the bandwidth is less than 8 Mb/s. Courtesy of JAI.For some companies active in machine vision, this focus is relatively new. For others, such as San Jose, Calif.-based JAI Inc., it is not. “A good 30 to 40 or 50 percent, depending on the way the year is going, of our North American business is in fact what you would put in the defense/homeland security area,” said Steve Kinney, director of technical presales and support.Faces in the crowdThat mix has been the case for years, and the company offers standard products that can be used in both vision and security applications. This is possible, in part, because of similar needs. For instance, both machine vision and security markets would like cameras with better light sensitivity. Consequently, there is a significant amount of technological back and forth between products intended for machine vision and those aimed at the security and defense market.Overlaying a visible video feed with data from a thermal imager reveals hidden information. Here an individual shows up as a white figure, properly placed in the overall scene. Courtesy of ICx Imaging Systems.Another example, Kinney noted, is resolution-proportional digital zoom, a patent-pending algorithm that maintains a constant data rate between a camera and host computer while digitally zooming the field of view. This trick is accomplished by proportional subsampling of pixels according to the zoom level.Employed in industrial settings, the application could enable verification of a part’s placement followed by zooming in to capture an identification mark. In surveillance, the technique could allow an airborne camera’s operator to spot a truck and then zoom in on its details, without overloading the data link.In general, better imaging across more wavelengths improves surveillance. For example, defense contractor ICx Imaging Systems of Apopka, Fla., offers products that combine video in the visible with thermal imaging. A thermal imager can determine whether an engine block on a car is hot, and a visible sensor can see inside rolled-up windows as well as read the vehicle’s license plate.Alone, neither capability provides the complete picture, noted ICx Imaging Systems’ general manager Nirav Pandya. However, when the two images are overlaid, all the information can be seen in one image. The company uses standard NTSC video cameras for its visible sensors, supplying expertise in the thermal imaging and data overlay arenas. Higher-resolution cameras capable of working in lower light conditions would add to the quality of the information, as would higher-resolution thermal imaging.One goal of surveillance is picking out faces in a crowd and at a distance, a feat that JAI’s Kinney said might be possible with a 4-megapixel, high-frame-rate camera. Such technology, however, also must account for current-day realities. Many surveillance applications require low-cost equipment and long cable runs between a central location and a camera. Today analog cameras are used, but GigE Vision-based systems might be used in the future. However, the infrastructure required for GigE Vision cameras — wiring, cameras and host systems — would increase the overall cost and complexity significantly.Even with better imaging capabilities, facial recognition in general settings is hampered by lighting and pose variables. Recently, researchers at the University of Glasgow in the UK found that averaging photographs together doubles facial recognition accuracy. About half of new photos of celebrities were correctly identified by FaceVACS, the face-recognition technology developed by the researchers. For averaged shots, on the other hand, the success rate jumped to 100 percent.Remote fingerprintsFacial recognition is one form of biometrics — measurements based on identifying unique biological characteristics. Fingerprints are the best-known and oldest biometric, but other body parts — such as the iris, retina and palm — could be used. The most basic of body parts, DNA, also has been pressed into biometric service.New technology allows fingerprints to be captured from a distance. A fingerprint captured from 30 in. (left) is compared with one taken from a standard ink print (right). The two were matched using the NIST algorithm BOSORTH3, and the agreement between 52 minutiae resulted in a very high confidence matching score. Courtesy of Northrop Grumman.With fingerprints, history is both a blessing and a burden. Large databases of fingerprints collected over the decades exist worldwide, making it possible to compare what is gathered currently at ports of entry or crime scenes against a large gallery. However, for such comparisons to work, what is collected today must mimic the prints made in the past by pressing an ink-stained finger against a piece of paper.That places special constraints on contactless digital fingerprinting, noted Erik Bowman, a principal engineer at defense contractor Northrop Grumman, based in Los Angeles. For example, you would need a full roll of a fingerprint, he said.The company has been working on a remote fingerprint system and hopes to have a prototype ready for use this summer. The device, imaging multiple fingers from 1 or 2 m away, will capture sufficient detail in a few seconds. This type of system would avoid the false features sometimes introduced by flattening the finger, and it also would circumvent the problem some people have with touching a surface previously handled by thousands.On the left is a raw 3-D image capture of a finger, whereas on the right is the transformation of that image into a standard 2-D print. Courtesy of Northrop Grumman.The system is based on standard machine vision technology, with a sensor of a few megapixels capturing the image. This type of contactless approach would mean that it could scan the iris or the face, for example, at the same time as it takes the fingerprints. Multimodal biometrics can resolve operational issues, including those arising from injuries or genetics. It can be hard to collect fingerprints from people who have had a serious hand injury or from someone with very fine fingerprints. Thus, multiple biometrics help increase the certainty of identification. However, the extra steps cannot cut into system throughput too much. Hence, it makes sense to determine as many biometrics at one time as possible.Daniel L. Lau, an associate professor of electrical and computer engineering at the University of Kentucky in Lexington, also has been working in the area of remote fingerprint capture. He uses structured light — illumination projected in a pattern — for example, striped. When the light falls across an object, distortions induced in the known projected pattern reveal three-dimensional information about the surface. For fingerprints, structured light allows the collection of extra information such as measurement of the depth of the ridges in the fingerprint, Lau noted. In their development work, Lau and his team have used a number of cameras, including industrial machine vision products from a variety of manufacturers, with megapixel sensors and connections over Gigabit Ethernet or Camera Link. The group also has evaluated the performance of the system when consumer cameras are used. These 12-mega-pixel devices that connect over a USB cable perform surprisingly well, according to Lau. Along with other engineers, he would like to get cameras that work better in low-light environments, citing the fact that full-system frame rates often cannot be achieved because integration time limits the camera speed to well below its maximum.Analyzing filmWith biometrics, the data often is a manageable single image. The same is not the case for video surveillance, which can churn out huge amounts of data quickly. A 4-megapixel camera taking 15 fps of 12-bit data will produce 96 MB of data per second. That translates to more than 300 GB/h.However, this data can be abstracted via video analytics. These techniques search for the movement of objects such as people walking the wrong way down a corridor. The methods also can detect when an object such as an unattended piece of luggage is left behind or when an item is removed or stolen. Video analytics are important because studies have shown that human operators quickly stop processing what is on a screen, perhaps in as little as 15 minutes.There are various approaches to remote fingerprinting. A camera and projector under the black plastic cover capture fingerprint details in three dimensions using structured light or patterned illumination. The distortions in the imaged pattern as compared with the illumination lead to the 3-D scan of the fingerprint on the screen. Courtesy of Daniel Lau, University of Kentucky.Avocado Security of Fairfax, Va., offers a different type of video analysis. The company’s software takes images from a security or surveillance system and converts them into metrics, such as the flow of people through a door or down an aisle. The system also can tell when a camera is malfunctioning, as the result of either a defect or of a deliberate attempt to disable it. The company’s software offers a number of reports based on information extracted from the video. What is generated follows the image-based version of the old computer maxim about garbage in, garbage out, noted the CEO of Avocado Security, Kevin Shahbazi. “If you give me a good image in, then I’m going to give you better results.”Better image quality is important, but those charged with security often face a dilemma. It is hard to justify throwing away an existing investment in cameras and wiring. That is one reason why the company’s software works with a wide variety of security cameras. It also is another reason why new systems based on the latest machine vision technology face some severe cost pressure when entering the security market.There also are other important considerations. Fabio Perelli, a product manager for Matrox Imaging of Dorval, Quebec, Canada, noted that the company shares technology between its product lines where possible. However, there are some cases where specific needs of the market force change, he noted. “One of the technologies we have created or adopted specifically for the security market is image authentication.”The technology authenticates down to the pixel level, a requirement if an image will be part of a criminal case; for example, when there must be clear proof that the picture has not been tampered with or altered.For the security market as a whole, he said that there is a trend toward higher-resolution and higher-frame-rate cameras, but this move is hampered by more than cost. There is little standardization, with many cameras following their own interface and control protocols. Eventually, that will change, but for now, this makes life difficult for vendors and end users alike. Matrox, for example, has to account for the various cameras it offers when making frame grabbers.Finally, there is a trend toward putting processing power in the camera, as is the case with the smart cameras found in machine vision. That potentially could solve the video problem by adding the power to analyze video to the camera, thereby cutting down the network traffic and the load on a host computer.“Smart cameras are potentially a good application for this, depending on the complexity of the analysis you like to do,” Perelli said.However, the task might be too much for the smart camera, he added. “There’s still a limited amount of processing power available to a smart camera compared to a rack-mounted PC, obviously.”