From the study of shrimp to DNA sequencing, high-speed cameras and imaging techniques are evolving to meet new applications in life sciences research.
JAMES SCHLETT, CONTRIBUTING EDITOR
Lurking among coral reefs, the snapping shrimp grows to be only ~3 cm, but it wields one of the most powerful claws under the sea. The creature’s claw can shut so quickly that it fires out a jet of water, generating a bubble that collapses to create the signature snap. But when those bubbles collapse, the cavitation also releases plasma and light energy. The snapper shrimp uses this effect to communicate or stun its prey. But its unique talent also generates underwater plasma more efficiently than electrically induced microbubbles or laser-induced cavitation bubbles. Efforts to mimic the crustacean’s biomechanics had fallen short until 2019, when researchers from Texas A&M University published a paper in Science Advances detailing how they had developed a bio-inspired device that could emulate the snapper shrimp’s talent for efficient plasma generation1. They achieved this with the help of ultrafast imaging.
A hummingbird in flight photographed with a high-speed camera featuring increased 2072- × 1536-MP resolution and throughput speeds up to 27.1 GP/s. Courtesy of iX Cameras.
Specifically, the Texas A&M team used a Photron camera to record the underwater operation of their bio-inspired device for the purposes of estimating its dactyl tip speed and to image the resulting cavitation at 60,000 frames per second (fps).
Beyond physical sciences
Shrimp studies are just one area in which ultrafast imaging is expanding beyond its core market in the physical sciences to explore opportunities in the life sciences. This imaging technology is now finding use in the study of DNA sequencing, turbulent flow dynamics, alternative drug delivery systems, nervous system and behavioral studies, and biomechanics.
“Ultrafast and high-speed cameras have cut their teeth in the physical sciences historically,” said Peter Carellas, president and CEO of iX Cameras. “They are now becoming more attractive to life scientists as applications involving imaging are being conjured. This natural evolution bodes well for ultrafast cameras as they replace conventional rate cameras in applications that now require very fast capture and processing of data.”
A sequence of three frames extracted from a video sequence of a mantis shrimp captured with an ultrahigh-speed camera capable of capturing 640- × 480-pixel images at frame rates >270,000 fps. This sequence shows that when the shrimp appendage hits the post, its force cavitates the water. Courtesy of RDI Technologies.
Photron, in 2010, introduced the camera used by the Texas A&M researchers, but a newer ultrahigh-speed iteration emerged in 2022, and is now being used by researchers at Duke University’s Patek Lab for similar cavitation studies on the mantis shrimp. This new camera can capture 640- × 480-pixel images at frame rates >270,000 fps.
Prior to acquiring Photron’s new ultrafast camera, the Duke lab had used a device with a maximum frame rate of 20,000 fps at 1-MP resolution, according to Tim Callenbach, Photron’s director of sales and marketing.
A frame taken from a high-speed video evaluating a mouse’s skilled reaching behavior. In life sciences experiments, modifications are made to the nervous system of a mouse to see how it affects the animal’s reaching behavior. A high-speed camera with synchronized LED strobe illumination captured the video at 720p (1280 × 720) and 1400 fps. At this frame rate, subtle changes in the motor control of the mouse can be observed and measured. Courtesy of Patek Lab at Duke University.
“Because the application requires the use of high magnification lenses (with lots of internal optics) and because the shrimp are in water and behind glass, and because a short exposure is required to slow the motion of the very, very fast-moving shrimp, light sensitivity is a problem,” Callenbach said. “Further, because the shrimp are so fast, it is necessary to utilize a very high frame rate.”
High speed versus ultrafast
Photron defines ultrafast as the ability to capture 70,000 fps at 1-MP resolution or ~250,000 fps at 640 × 480 pixels. In contrast, high speed requires a capture capacity of only 2000 fps at a minimum resolution of 1 MP. According to Callenbach, high-speed cameras offering resolutions of 3, 4, and 5 MP are becoming more commonplace; and 9.4-MP high-speed cameras — known in the industry as 4K cameras — are now available with frame rates >1000 fps.
Ultrafast photoactivation/photoconversion localization microscopy (PALM) images captured with a single fluorescent-molecule imaging system. The researchers developed the system to analyze single-molecule trajectories in plasma membranes and demonstrated that it could image a fluorescent dye at a frame rate of 30 kHz, or 1000× faster than conventional video. Here, it reveals the shape changes, migrations, and formation/disappearance of caveolae in live cells.
Diffraction-limited and PALM images of caveolae (a) in 10- × 10-µm2 observation areas, using a data acquisition period of 1 s. The purple
outlined images in the middle column (b) show expanded images of the purple-square regions in (a) for data acquisition between 1 and 2 s.
The regions surrounded by yellow squares in the middle row are magnified in the bottom row, and the localizations of single mEos3.2 molecules
determined within each 1-s period are indicated by the red dots (c).
Courtesy of Okinawa Institute of Science and Technology Graduate University.
As sensors achieve faster pixel throughput rates and new and improved backside illumination (BSI) designs emerge, high-speed cameras will advance into realms previously reserved for ultrafast imaging. “Although traditional ultrafast cameras continue to have significantly higher frames rates — at least for now — they are ceding market space due to their limitations in the numbers of frames they can capture and in their somewhat limited light sensitivity,” Callenbach said.
Speed limitations
Both ultrafast and high-speed videography obviously demand short exposure times, which often benefit from additional illumination of imaged subjects via
external lighting. The company iX Cameras has leveraged recent advancements in LED lighting, low-cost laser illumination, high-powered narrow-band illumination,
and structured lighting solutions to develop a patent-pending synchronized integrated light control technology aimed at complementing nanosecond-scale exposure times.
Lighting is also important when optimizing the relationship between shutter speed and resolution. This linkage is often less of an issue with conventional cameras. But for ultrafast imaging, image resolution is linked to frame speed and frame speed helps determine shutter speed. Ultimately, as frame rate and shutter speed increase, the sensor has less time to collect photons. “This highlights the need for innovative lighting and associated lighting control as a feature,” Carellas said.
RDI Technologies has also developed synchronized LED strobe illumination to address many different industrial and scientific applications. Facilitating short exposure times and wider lens apertures, the LED strobes produce bright, short pulses of light that are synchronized with the camera shutter and frame rate. “The result is sharp images without motion blur and greater depth of field with frame rates over 16,000 fps,” said Tom Boldt, the sales manager for RDI’s Fastec brand.
Increasing frame rate not only requires faster integration of photons into the pixel, it also demands faster readout and processing. “As sensors become more powerful and are able to process more pixels per second, the collection of technologies — whether streaming-over-Ethernet or storing locally on the camera — is now limiting the camera speeds,” Carellas said. “New image/data transfer technologies, faster FPGAs [field programmable gate arrays], and denser memory are all examples of how ultrafast camera designers are innovating in this space.”
Cellular processes
Microscopy studies that aim to capture cellular processes use extremely high-speed imagers based on scientific CMOS (sCMOS) sensors. Such devices deliver low noise, wide 16-bit dynamic range, and high quantum efficiency. Capturing such brief events can require selective compromises to find the right balance for the target application.
Caltech’s compressed ultrafast spectral photography (CUSP) system achieved a new imaging speed record of 219 trillion fps. Courtesy of Caltech.
Permitting more noise was key to achieving faster rates in an experimental single fluorescent-molecule imaging (SFMI) system developed by an international team that included Okinawa’s Institute of Science and Technology Graduate University and the Institute for Integrated Cell-Material Sciences in Kyoto. The researchers developed their system to analyze single-molecule trajectories in plasma membranes and demonstrated that it could image a fluorescent dye at a frame rate of 30 kHz, or 1000× faster than conventional video. Previous SFMI cameras had only achieved 400-Hz speeds.
“I think, previously, scientists and engineers working on SFMI were preoccupied by the notion that developing a sensor with lower readout noise is the key issue. Probably, they never thought of using faster, but much noisier CMOS sensors,” said Akihiro Kusumi, professor of biophysics at the Okinawa Institute of Science and Technology Graduate University.
While ultrahigh-speed scientific imagers usually use sCMOS photosensors, Kusumi said his team used a more conventional CMOS chip that, in most cases, would be problematic for single-molecule imaging because its readout noise is larger by a factor of several hundreds. “We needed to use the CMOS sensor for its speed,” Kusumi explained.
Leveraging a technique gleaned from nonlinear electron paramagnetic resonance spectroscopy, the team overcame the large readout noise of its CMOS sensor by employing an image intensifier with a three-stage microchannel plate as a preamplifier. “Since the noise generated at the final detector (sensor) is large … both the signal and noise generated before the sensor stage could be amplified without worrying about making the noise level too great at earlier stages,” Kusumi said.
Consequently, the spatiotemporal resolution of the system is photon-limited by fluorophore photophysics rather than the camera. The camera can be operated at up to 110 kHz, but the precision of single-molecule localization becomes unacceptably low at that speed due to the limited number of photons obtained from a single fluorescent molecule. With its SFMI system, the team could track extremely brief dynamic movements and molecular interactions. For example, they were able to observe molecules in a cellular plasma membrane undergo hop diffusion even within the focal adhesion, according to Kusumi.
Another record-breaking ultrafast imaging advancement for the life sciences was developed by California Institute of Technology’s Optical Imaging Laboratory. In 2020, the Caltech team debuted a compressed ultrafast spectral photography (CUSP) system able to register 70 trillion fps. According to the team, the system was the first-ever single-shot spectrum-resolved fluorescence lifetime imaging microscope (SR-FLIM) able to record picosecond-scale maps with good spatial and spectral resolutions in a single acquisition. Last year, the Caltech team revealed it had renovated its CUSP system to set a new imaging speed record of 219 trillion fps, achieved by optimizing the chirp parameter in the chirped pulse train.
The CUSP system combines extremely short pulses from a Coherent femtosecond laser and high-reflectivity low-chirp
ultrafast mirrors with a Hamamatsu streak camera delivering a shearing speed of 10 THz with a time range of 50 ps.
“Instead of using temporal shearing by the streak camera to distinguish time, we exploited a rich number of wavelengths in a chirped ultrashort laser pulse to stamp time at a much finer resolution. The imaging speed of CUSP is no longer bound by the electron sweeping speed in the streak camera. To overcome the issue of limited observation windows, we introduced a pulse train to extend the total number of frames. As a result, nearly 1000 frames can be captured in a single shot,” said Lihong Wang, a Caltech professor of medical engineering and electrical engineering, who led the CUSP project with the help of Peng Wang, a postdoctoral student.
The cost of speed
Life sciences researchers are still learning what high-speed imaging can achieve for them, said Patrick Summ, a managing sales director for Optronis. This leads them to either overestimate what high-speed imaging can do or else overlook its possibilities, he said.
One factor behind that overestimation is that many high-speed and ultrafast camera setups are too costly for many life sciences researchers. They often range anywhere from $20,000 to $250,000. Caltech’s CUSP system costs more than double that high-end price, according to Wang.
Noting how the performance of cameras within that price range continues to improve as technology improves, Callenbach believes that cameras priced at the lower end will continue to improve to the point where they will find broader use within all of the scientific disciplines that can benefit from high-speed camera implementation.
References
1. X. Tang and D. Staack. (2019). Bioinspired mechanical device generates plasma in water via cavitation. Sci Adv, Vol. 5, No. 3, eaau7765.