Medical spectral imaging cameras built into endoscopes scan multiple wavelengths, empowering diagnostics and therapeutics in specialties ranging from airway management to cardiology.
TEHZEEB GUNJA, OMNIVISION, AND AXEL KULCKE, DIASPECTIVE VISION
Upgrades in medical imaging
technology have made possible the
accurate diagnosis and successful
treatment of ailments ranging from
broken bones to cancer. Techniques such
as endoscopy keep evolving to image at
greater depth and with stronger resolution,
and advancements in miniaturization,
electronic sensing, and computing
capabilities inform the decision-making
of clinicians. Physicians have an ever-growing
array of imaging modalities
available to them — such as hyperspectral
techniques — that enhance their ability
to treat injury and disease with greater
clarity and speed and with less patient
discomfort.
Intraoperative
diagnosis, interventional
therapy, and minimally invasive surgery
are possible because of advancements in
endoscopy. The instrumentation and use of
endoscopy have evolved over the decades
as high-resolution imaging has become
more adaptable. In modern medicine,
the technique can do much more than
visualize areas of the body that are easy to
access through natural orifices. The range
of applications in which endoscopy has
proved useful include gastroenterology,
airway management, gynecology, arthroscopy,
and cardiology.
Extremely small image sensors using
complementary metal oxide semiconductor
(CMOS) technology have been embedded
on the tips of endoscopes, allowing
cameras to enter the body through ever
smaller incisions1. State-of-the-art software
and hardware for image processing
enable high-speed video that serves as the
“eyes” of the surgeon as he or she works
inside the body (Figure 1). Surgeons can
then see where to cut in real time.
Figure 1. An endoscope
and an image it provided.
Courtesy of OMNIVISION.
Common imaging techniques
Diagnostic imaging has traditionally
been grouped into four categories of tools:
x-ray, including both single-image and
computed tomography (CT) scans; magnetic
resonance imaging (MRI); nuclear
medicine with radioactive tracers, including
positron emission tomography (PET);
and ultrasound (Figure 2).
These techniques are relatively noninvasive
and are valuable tools for helping
physicians decide on a course of treatment.
However, the tools carry risks from
exposure to ionizing radiation, and some
patients experience allergic reactions
to injected tracers. Also, most of these
imaging techniques cannot visualize the
area of interest intraoperatively. Physicians
therefore use the images as more
of a reference to guide treatment, not as a
real-time aid.
Figure 2. An example image from each of the four traditional categories of diagnostic imaging: CT scan (a), MRI (b), PET (c), and ultrasound (d). Courtesy of
OMNIVISION.
In this context, endoscopy is a wide-ranging
technique with many advantages
over traditional imaging technologies
employed in conjunction with surgery.
Robotic surgical instruments have
advanced the uses of endoscopy even
further. Available tools are much more
flexible than a surgeon’s hand and can be
directed precisely at the point of interest.
But despite the advancements in endoscopy,
limitations still exist in the information
it captures. The endoscope does not
necessarily offer the surgeon a full 360º
view of the area of interest. In most cases,
imaging covers only the visible portion of
the spectrum and, as a result, distinguishing
between different types of tissue can
be difficult.
Overall, the benefits of procedures
guided by endoscopy outweigh the risks
when compared to traditional surgery,
which often requires large incisions. Patients
with less invasive operations spend
less time in surgery and recover faster.
But risks remain even if endoscopy is
used. Wounds can heal less quickly when
oxygenation is cut off from remaining
tissue. Recognizing underperfused tissue
— where circulation is poor — requires
a judgment call based on a color change
in the tissue. Additionally, in an effort to
avoid resecting healthy tissue, endoscopic
surgery sometimes leaves behind some
of the cancerous tissue not visible to the
human eye. The only way to concretely
confirm that the cancer is gone is through
follow-up medical imaging after surgery.
Enhanced vision
Fortunately for clinicians and their
patients, a new imaging solution could potentially
transform endoscopy and expand
its capabilities. Hyperspectral imaging
utilized in state-of-the-art endoscopic
cameras holds the promise of higher
productivity of detailed images, enabling
the thoroughness of surgical intervention
along with lower risk of damage to a
patient’s collateral structures.
Physicians use endoscopes of various
types to see clear images of tiny
structures inside the human body. With
this guidance, clinicians can accurately
diagnose disease and can often treat it immediately
using endoscopic surgical tools.
But what if these tools could help doctors
distinguish between nerves, veins, and
muscle tissue? Or what if a doctor could
see the level of perfusion and, along with
it, the viability of the tissues’ healing
capability?
From the viewpoint of a standard endoscopic
camera, organ tissue looks red and
has a distinct texture. But there is no way
to specify embedded structures and blood
circulation in the resulting images.
One way to visualize perfusion is by
injecting the patient with indocyanine
green (ICG). When the use of ICG is combined
with the insertion of an endoscope,
perfusion visualization is possible to a
limited degree within a subject’s systems.
The color agent allows the physician to
better assess the location of the tissue that
needs to be removed and which areas to
avoid in the subsequent surgery. The timing
of ICG injection is tricky, however, as
fluorescence is present only for a limited
time, making the method less effective for
lengthy surgeries. At the same time, repeated
color injections are likely to result
in poor contrast because of the accumulation
of residual tracer left in the body.
It is possible to imagine even greater
enhancements to endoscopy as research
and development continue. What if tumors
showed up as distinct from healthy
tissue in images, so that oncologists could
literally see cancer and remove it on
the spot? The creation of a such a clear
picture is not yet possible in medical procedures,
but it is also not science fiction.
What is spectral imaging?
Medical spectral imaging, an umbrella
term for hyperspectral imaging, is a new
application of an imaging technique that
has existed for decades in other industries.
The technology combines absorption
spectroscopy and digital imaging. It is applicable
for use in a wide range of fields,
including mining, agriculture, military
combat, and medicine.
Within the field of view, hyperspectral
imaging scans multiple wavelengths of
light over a range of 400 to 1000 nm,
thus extending from visible light into
the near-infrared (NIR) region of the
electromagnetic spectrum. The longer the
wavelength, the deeper into the sample
the light can penetrate.
Each wavelength of light will interact
differently with the particular material
that the light is directed toward, depending
on the material’s chemical composition.
The amounts of oxygen, water, or
other specific molecules in the sample
affect the degree to which light can be
absorbed by the materials. Converting the
absorbance value into a color creates a
visual representation of the results (contrast)
in image form.
A high-resolution camera processes
data from millions of individual pixels in
a sensor. The spectrometer captures the
spectral data from each pixel, resulting in
a cube of images that represents two spatial
dimensions (x and y) plus wavelength
(λ)2. Imaging software enables the user to
see physiological differences in the tissue
in the camera’s field of view.
A new type of camera
The capabilities of medical spectral
imaging are well suited to clinical applications.
The various tissues in the human
body have a unique spectral signature that
traditional imaging technologies such as
ultrasound cannot capture. But combining
hyperspectral imaging with other imaging
modalities expands their efficacy.
Diaspective Vision has taken advantage
of this synergy in technical capability
to incorporate medical spectral imaging
into a new type of endoscopic camera, the
MALYNA system. When applied to high-resolution
medical imaging applications,
this unique combination of technologies
can expand the benefits of endoscopic
procedures (Figure 3). The camera is applicable
in laparoscopy, where it has seen
clinical use, and it is also under development
for robotic surgery and diagnostic
endoscopy. The system augments 4K live
video streaming along with physiological
information to provide surgeons with
visual confirmation that can inform their
decision-making in real time.
Figure 3. The MALYNA system incorporates medical spectral imaging into a new type of endoscopic
camera. Courtesy of Diaspective Vision.
The benefits of using this advanced
imaging technology include higher productivity,
reduced patient risk, and better
post-surgery outcomes because the tissue
affected by a particular condition or
procedure is easily identifiable. Hyperspectral
imaging capabilities enhance the
performance of even the most advanced
standalone endoscopic cameras. The success
of medical spectral imaging requires
leveraging the most advanced components
available for high-end endoscopes. And it
requires high-performance CMOS image
sensors with 4K resolution and low latency
for streaming video at 30 or 60 fps.
Most endoscopes incorporate sensors
that operate in the visible spectrum, but
these sensors are not sufficient for medical
spectral imaging. NIR light-sensitive
image sensors are necessary to take advantage
of the expanding imaging power.
The sensors must be medical grade, with
high quantum efficiency covering the
extended spectral range of the system.
While image sensors that detect NIR
light have been commercially available
for some time, they are usually associated
with applications such as security
that need to record images in low-light
conditions. The same technology that allows
security cameras to see intruders at
night, however, can be applied to medical
applications. Medical-grade sensors with
OMNIVISION’s Nyxel technology, which
increases sensitivity in the NIR region,
meet the necessary requirements. Medical
imaging cameras are now capable of distinguishing
between tissues with different
spectral signatures in the NIR range.
Using medical spectral imaging
Perfusion visualization is a key application
for which medical spectral
imaging offers significant benefits. The
new camera technology supports ICG-based
perfusion visualization, but it goes
further, offering quantified perfusion data
without the need for any color agents. The
system can convert real-time measurements
of oxygen content in the blood into
a live video stream that shows the area of
interest in full color.
This increased technological capacity
enables physicians to examine perfusion
at any time, regardless of the duration of
the procedure. There is no need to delay
the next step while waiting for ICG to dissipate
in a person’s system.
When a surgeon is repairing a wound,
real-time perfusion visualization allows
him to optimize the repair to promote
self-healing. Medical spectral imaging can
also show the location of nerves, veins,
and arteries that need to be avoided during
surgery. As a result, surgeons avoid the
risk of accidentally cutting a nerve and
causing the patient permanent damage.
NIR imaging goes deeper into tissue
than imaging in the visible spectrum
does. With NIR sensors, images can
be captured from up to 6 mm under
the surface, allowing physicians to see
underlying structures before making any
incisions, which improves the accuracy of
endoscopic surgery by limiting incisions
to an appropriate area.
Medical spectral imaging can quantify
oxygenation and provides indexing
of hemoglobin water and lipid content.
Surgeons can use the data to identify
various types of tissue as they move the
endoscope (Figure 4)3.
Figure 4. The visualization modes of the MALYNA endoscopic camera system. Courtesy of Diaspective Vision.
Potential future applications
Each type of tissue and each organ in
the human body has a unique spectral
footprint. With advancements in deep
learning software and image mapping,
it should be possible to distinguish even
more types of tissue, including nerves and
head and neck tumors. Embedded features
of specific tissues that were previously
invisible to a clinician for diagnostic
purposes will become visible and inform
a growing number of procedures.
If the medical spectral imaging system
can identify particular organs, robotic
surgery will be an option for a greater
variety of procedures. In the near future,
it may even be possible to distinguish
clusters of cancer cells using hyperspectral
imaging technology in conjunction
with deep learning methods. This capability
could potentially make tumor removal
more precise, protecting noncancerous
areas. Scans after surgery could then
verify whether the cancer was completely
removed before the end of the procedure.
Medical spectral imaging, in combination
with state-of-the-art endoscopic
cameras, is set to become a standard
diagnostic tool, alongside contemporary
modalities such as x-ray, MRI, nuclear
medicine, and ultrasound. Unlike the
legacy tools, however, hyperspectral
imaging can support both diagnosis and
treatment of injuries and diseases, leading
to better outcomes for the patient and
quicker recovery times.
Meet the authors
Tehzeeb Gunja is director of medical marketing
at OMNIVISION. He holds a Bachelor
of Science degree in electronics from the
University of Mumbai and a Master of Science
degree in electrical engineering from Wayne
State University; email: [email protected].
Axel Kulcke, Ph.D., founded Diaspective
Vision GmbH in 2015 after studying physics
and chemistry at Georg August University of
Göttingen, followed by earning a doctorate at
the same university and working in various
positions in industries adopting spectral imaging
technologies; email: [email protected].
References
1. R. Yang (2021). Chip-on-tip technology
expands endoscopy’s use in localized
procedures. BioPhotonics, Vol. 28, No. 1,
pp. 30-36, www.photonics.com/articles/chip-on-tip_technology_expands_endoscopys_use_i/p1/vo201/i1267/a66501.
2. P. Heney (2020). What is hyperspectral
image analysis? R&D World, www.rdworldonline.com/what-is-hyperspectralimage-analysis.
3. B. Jansen-Winkeln et al. (2019). Determination
of the transection margin during
colorectal resection with hyperspectral
imaging (HSI). Int J Colorectal Dis, Vol. 34,
pp. 731-739, www.doi.org/10.1007/s00384-019-03250-0.