Stephen Hawking once said, “Remember to look up at the stars and not down at your feet.” This deep-seated urge to make sense of where we came from and to try to understand what makes the universe tick is a defining aspect of what makes us human. Combining high-resolution, multispectral capabilities, a swath width of 290 km, and frequent revisit times, the Copernicus Sentinel-2 mission is designed to play a key role in mapping differences in land cover to understand how the land is used and to monitor changes over time. It is being developed by ESA for the European Global Monitoring for Environment and Security program. Courtesy of Astrium GmbH. In addition to providing detailed information about Earth’s vegetation, , the Copernicus Sentinel-2 mission is designed to play a key role in mapping differences in land cover to understand the landscape, map how it is used, and monitor changes over time. Courtesy of ESA/ATG medialab. Another uniquely human quality is self-assessment: directing our view inward or, as a species, downward from above to gauge the impact we have on our surroundings. Observations from above can be carried out by unmanned aerial vehicles (UAVs), piloted aircraft, or constellations of satellites that keep us informed about the changing composition of the land, sea, and air around us. Uppermost in elevation — and cited as a “game changer” by some Earth observation experts — is the European Union’s Copernicus program. Implemented by the European Space Agency (ESA), the program provides data from a growing number of satellites. The satellite missions, called Sentinels, will eventually comprise 30 spaceborne platforms flying in constellation to provide high-quality image data. Green algae blooms swirl around the Baltic Sea. The Baltic Sea faces many serious challenges, including toxic pollutants, deep-water oxygen deficiencies, and toxic blooms of cyanobacteria affecting the ecosystem, aquaculture, and tourism. Sentinel-2 image contains modified Copernicus Sentinel data (2019). Courtesy of ESA/CC BY-SA 3.0 IGO. Although it may be mistaken as an abstract watercolor painting, this image from the Sentinel-2A satellite captures the natural color of a plankton bloom in the Barents Sea. Plankton are able to convert inorganic compounds such as water, nitrogen and carbon into complex organic materials. With their ability to ‘digest’ these compounds, they are credited with removing as much carbon dioxide from the atmosphere as their counterparts on land. As a result, the oceans have a profound influence on climate. Since plankton is a major influence on the amount of carbon in the atmosphere and are sensitive to environmental changes, it is important to monitor and model them into calculations of future climate change. Image contains modified Copernicus Sentinel data. Courtesy of ESA/CC BY-SA 3.0 IGO. Data from the Sentinel project is free for anyone to download at www.coperni cus.eu/en/access-data. Sentinel-2 images Earth in the optical domain to provide 12 TB of data every day. The data from this mission contributed to 700 articles in peer-reviewed journals in 2019 alone, making Sentinel-2 the most cited European Earth observation mission. For those assessing challenges such as deforestation, overfishing, and soil erosion, the Copernicus program offers six services dedicated to monitoring various components of the Earth system — such as land, atmosphere, and the marine environment — as well as services dedicated to climate change, emergency management, and security. “Earth is a living planet that is changing all the time,” said Ferran Gascon, Sentinel-2 Mission manager at ESA. “Therefore, observing Earth needs to be done systematically across the whole globe so we don’t miss events like wildfires and floods, but also to carefully monitor changes in the health of Earth’s vegetation and issues such as deforestation.” Sentinel-2 comprises a constellation of two satellites that image our entire planet every five days. Their routine acquisitions allow the mapping of threats such as deforestation, and they support countermeasures to prevent the threats’ spread. This is particularly important for remote and inaccessible regions, such as Amazonia, which has recently suffered swaths of fires, large-scale deforestation, and land grabbing. From population density mapping to security and defense applications, Sentinel-2 data can also monitor traffic patterns; characterize the speed, altitude, and direction of aircraft; and measure ground motion resulting from earthquakes. On board each satellite is an identical multispectral instrument that can capture image data in 13 spectral bands spanning the visible, near-infrared (NIR), and shortwave infrared (SWIR) segments of the electromagnetic spectrum. Images are provided at spatial resolutions of 10, 20, or 60 m, depending on the spectral band and the image’s intended use. Light reaching the multispectral instrument from Earth is collected by a three-mirror anastigmat telescope and directed, via a beamsplitter, onto two focal plane assemblies: one for the 10 visible and NIR spectral bands and one for the three SWIR spectral bands. Taking advantage of the forward motion of the spacecraft along its orbital path furnishes the multispectral instrument’s pushbroom sensor with new rows of image data across a 290-km orbital swath, which builds to form a 2D image. Beyond land imaging, Sentinel-5P — the latest satellite to launch as part of the Copernicus series — is charting air pollutants around the world. “Air pollution is a global environmental health problem that is responsible for millions of people dying prematurely every year,” Gascon said. “With air quality a serious concern, optical missions such as Copernicus Sentinel-5P allow the mapping of a multitude of air pollutants such as nitrogen dioxide around the globe.” Down-to-Earth interests In addition to environmental concerns, Earth imaging is an increasingly useful tool for government functions overseeing environmental health, public works, community development, and tax assessment. High-resolution imagery captured by sensors mounted on aircraft delivers resolution between 2 and 30 cm GSD (ground sampling distance) and positional accuracy up to 2× GSD. Such resolutions are sufficient for detailed analysis of erosion and flood zones, as well as for the support of city planning and power line maintenance. An area in the Santa Cruz Department of Bolivia, where part of the tropical dry forest has been cleared for agricultural use. Sentinel-2 image contains modified Copernicus Sentinel data (2019). Courtesy of ESA/CC BY-SA 3.0 IGO. By capturing data from an aircraft, data providers can guarantee the collection of entire states or countries in one season, providing homogenous data sets that are ideal for machine learning and analytics. Courtesy of Hexagon AB. Though airborne imaging requires more advanced planning, it can typically capture data for an entire state within one season and provide highly consistent data. Stockholm-based Hexagon AB, via its HxGN Content Program, provides orthorectified images of the U.S., parts of Europe, and areas of Canada to a resolution of 30 cm, with 15-cm coverage over metropolitan areas encompassing populations greater than 50,000. Those who subscribe to the program can access an online map that includes rich layers of data about zoning, fire and school districts, roads, emergency services, and channel migration zones overlaid on high-resolution imagery. Emergency responders such as police and firefighters are aided by the address point layer to pinpoint the fastest way to reach a person in need, while engineers in the private sector can more easily design septic systems by analyzing online data. For shoreline properties, developers can check for extra regulatory restrictions to ensure compliance with proximity to shoreline and erosion concerns. Investments in lidar The push to develop self-driving cars has led to more compact and power-efficient lidar systems that are now finding application aboard aerial platforms, which can combine high-resolution imaging with lidar data in a single flight. A false-color image of Southern Ukraine, captured in the near-infrared by the Sentinel-2 mission and processed to show vegetation as bright red. Image contains modified Copernicus Sentinel data (2019). Courtesy of ESA/CC BY-SA 3.0 IGO. In recent years, Hexagon began developing hybrid technology sensors that simultaneously collect data in the form of both imagery (oblique and nadir) and lidar height/intensity data. “This will be the trend for the future for airborne sensing and large data programs,” said Ron Roth, project manager for Airborne Topographic LiDAR at Hexagon’s Geosystems Division. An orthorectified hyperspectral imaging swath taken by a Headwall UAV using 30-m resolution U.S. Geological Survey (USGS) digital elevation model (DEM) data (top). Orthorectification of the same hyperspectral imaging swath using 10-cm resolution DEM data obtained by using the integrated lidar (bottom). The distortions in the first image are absent in the second. SWIR image with red = 1597 nm, green = 1196 nm, and blue = 1597 nm. Courtesy of Headwall Photonics Inc. The Hexagon Content Program’s coverage spans the contiguous United States, Europe, and urban centers in Canada to capture airborne image data at very high resolution. It represents largest collection of orthophotos collected with aircraft-mounted sensors. Courtesy of Hexagon AB. In hybrid sensors, imaging is carried out by medium-format frame-type cameras, such as the MFC150 modules used inside Leica’s CityMapper 2. This technology is used specifically for 3D urban mapping as opposed to the wide-area capture currently acquired in imaging content programs. However, there is future potential for 3D city models, including oblique images, to become an additional layer in content programs. ‘These medium-format cameras provide image resolutions that would have been considered “large format” only a few years ago.’ Airborne frame-type imaging systems typically employ forward motion compensation, which adjusts the focal plane in the flight direction to cancel any image motion and the resultant blur while the shutter is open. “In addition to RGB cameras, near-IR images can also be collected with a separate camera fitted with a near-IR focal plane,” Roth said. “These medium-format cameras provide image resolutions that would have been considered ‘large format’ only a few years ago, but are our current medium-format products.” Hexagon also manufactures both linear-mode and single-photon lidar systems. Both approaches employ short laser pulses to determine distance to the ground as well as surface reflectivity. A color-infrared image of Port Townsend in Jefferson County, Wash., enables the study of vegetation that appears in bright red. Remote imaging data also provides county officials with rich data layers to govern zoning, road maintenance, and emergency services. Courtesy of Hexagon AB. A 15-cm-resolution orthophoto of a San Francisco neighborhood showing the many challenges that can be managed from the air, including vegetation encroachment, solar potential, parking capacity, and property assessments for insurance. Courtesy of Hexagon AB. This high-resolution aerial image of the Port of Los Angeles harbor shows how much detail and information can be extracted from 15 cm orthophotos. This resolution can only be captured with aircraft-mounted sensors over large areas. Courtesy of Hexagon AB. “This laser range-finding device is scanned across the terrain below the aircraft, while the instrument inside the aircraft continuously measures the aircraft location and orientation with respect to the Earth below,” Roth said. “Processing that source data allows the generation of a point cloud, where each firing of the laser provides a position (latitude, longitude, and elevation) and intensity value, including for multiple return reflections that happen when flying over forests, at the edges of buildings, and over power lines.” In linear-mode technology, the laser pulses are robust, and single detector elements respond to the incoming reflections within the so-called linear zone. Such systems are ideal for topographic bathymetry, as well as shallow- and deep-water bathymetry. In single-photon technology, the laser pulses are much shorter, with a single laser output split into 100 “beamlets” and the return reflections falling onto a 10 × 10 array of extremely sensitive detectors, each capable of detecting as few as one or two photons. “Our single-photon lidar sensor can collect 6 million points per second, enabling efficient lidar data collection of entire countries,” Roth said. Spectral imaging specialist Headwall Photonics in Bolton, Mass., can capture spectral and lidar data over any terrain, for fast feedback on the state of anything from vegetation and rock composition to the soundness of buildings and railroads. Although the company is a leader in spectral imaging, the benefits of using lidar are becoming more well known, and the belief is that both technologies will evolve together to become more pervasive as researchers and other end users become aware of how the technologies can be used together. In fact, Headwall’s sensor platforms can embed lidar visualization data into the hyperspectral images, enabling critical visualization opportunities for end users. “We believe that hyperspectral imaging and lidar are not competing technologies but rather complementary technologies,” said Ross Nakatsuji, senior product manager at Headwall Photonics. “In fact, we were the first company to offer an out-of-the-box lightweight UAV with both hyperspectral imaging and lidar together as a standard product offering.” While lidar is great at collecting data on surface positions on the ground or on objects such as buildings, rocks, crops, or trees, hyperspectral imaging adds the spectral signatures of the same objects. This data can provide a wealth of intelligence gathered from above, such as determining whether a building is suffering from pooling water, rocks contain certain minerals, crops are being optimally fertilized, or trees are healthy. Reducing data costs Imaging is a demanding discipline. No sooner has the latest technology demonstrated itself in an application than customers are clamoring for the next step up. This could be higher resolution, more data points, faster processing, or more sophisticated interpretation of the data. “A coming challenge for [the] Sentinel- 2 mission is to meet the growing needs of the users. For instance, users need a higher spatial resolution, higher temporal revisits to increase the chances of capturing land surface changes, and a higher number of spectral bands to enable better characterization of the targets,” Gascon said. “All of these needs are currently being considered in the context of preparing for the future and ensuring a continuity of the Sentinel-2 mission during [the] next decade and beyond.” The demanding nature is universal across the industry, and by pursuing hybrid sensor solutions, companies such as Hexagon and Headwall Photonics can provide not only more information per data point but also more data points per second. This results in making fewer flights, which serves to minimize aircraft operation costs, and reduces processing costs. “There is a limited ‘world budget’ for acquiring the data we need to collect geospatial intelligence and support decisions at both local and global levels,” Roth said. “In addition, the world is changing faster, meaning that we need this data updated more frequently. The clash between these two needs can only be resolved by increasing the efficiency of data acquisition, processing, and delivery.” Because most of this data is collected from costly air-based platforms — where operating costs dominate the cost of overall data delivery — collecting more data at each point in space and time makes sense. Imaging the COVID-19 Outbreak Notoriously dynamic and quick to respond, the photonics industry proves its versatility and broad applicability yet again as it pulls together to not only capture the effects of the COVID-19 lockdown but also to show how some regions are beginning to recover. Italy’s effort to limit the spread of the coronavirus disease has led to a decrease in boat traffic in Venice’s famous waterways. Data captured by the Copernicus Sentinel-2 mission shows the effect on the locked-down city of Venice in northern Italy. Boat traffic on April 13 of this year (top) compared to that of April 19, 2019 (bottom). Image contains modified Copernicus Sentinel data (2019-2020). Courtesy of ESA/CC BY-SA 3.0 IGO. Emergencies don’t get much bigger than the COVID-19 outbreak. In response, the European Space Agency (ESA) and the European Commission jointly developed a Rapid Action Coronavirus Earth (RACE) observation dashboard (https://race.esa.int) to leverage remote satellite images and shed light on the societal and economic changes underway. Using Earth observation data coming from the Copernicus Sentinels and ESA third-party missions, the dashboard not only captures the effects of the lockdown, but also shows — via a set of economic, agricultural, and environmental indicators — how Europe is beginning to recover. This information is made available to all Europeans so that they can assess the impacts of COVID-19, in the hope that the data will eventually foster detailed analysis on a regional scale by public or private administrations. In similarly cooperative fashion, Hexagon recently made the entire imagery library from its HxGN Content Program freely available to any government agencies and nonprofit organizations managing the COVID-19 outbreak. To date, many national, regional, and local agencies and their supporting contractors have used the program data to track, visualize, and plan their response efforts to the pandemic.