“Fight like you train and train like you fight” is a military maxim. To better do that, the military is turning to virtual, augmented and mixed reality. In virtual reality (VR), digital representations replace the real world, with a high-end example being immersive simulators with realistic sights, sounds and motion. In augmented reality (AR), projected data overlays what trainees see. Mixed reality (MR) combines the real and synthetic. When there are multiple projectors in a virtual reality simulation, they must have their output blended and warped to avoid artifacts where outputs overlap. Here is the result of a camera-based blending and warping. Courtesy of Electric Picture Display Systems. Substituting representations for actual friend, foe, equipment and hazards enables less costly, safer and more frequent training of airborne pilots, seagoing navigators, maintenance and repair technicians, and others. In this, VR/AR/MR benefit from ongoing display improvements. “As technology advances, more capabilities are possible, and therefore it’s possible to provide higher-fidelity solutions,” said Benito Graniela, visual systems engineer at the Naval Air Warfare Center Training Systems Division in Orlando, Fla. Gains in resolution, luminance In general, costs are down and performance up. Consumer industry demand has cut the cost of computing platforms, graphics cards, memory and disk drives. Also, display aspects such as resolution, contrast, luminance, refresh rate and geometric distortion have gotten better. That makes critical visual cues appear more lifelike. Nine projectors installed 32 in. above the floor create a 360-degree image, used in virtual reality simulations for military and other training. Courtesy of Electric Picture Display Systems. “The economies of scale generated by these markets and the shift from custom build to COTS (commercial off-the-shelf technology) have allowed for higher-fidelity virtual environments at the same cost of systems of five or 10 years ago,” Graniela said. The recent development of inexpensive yet capable head-mounted units promises continued future improvement. However, more needs to be done, such as in deploying higher resolution and faster refresh devices. The result could eventually be a low-footprint yet highly immersive experience. At the same time, though, as important as the display are the seamless integration of the virtual environment, the simulation models and the human-computer interface. Furthermore, simulation or motion sickness has been the primary reason why such applications have not advanced further, although work has been done to solve some of the basic problems, Graniela said. An engineer with the Naval Air Warfare Center Training Systems Division tests a commercial off-the-shelf (COTS) virtual reality headset. Such headsets could help in virtual reality based training, particularly after further development and improvement. Courtesy of U.S. Navy. In a large system, the scenes displayed are constructed via multiple projected images, which present challenges. The different individual segments, for example, must be seamlessly blended together. In military applications, this can be difficult because training may involve simulating operations at night. Hence, not only must the display be bright enough but the contrast must also be high. It may perhaps have to be as much as two million to one for simulating the nighttime landing of Navy planes on aircraft carriers. While system resolution is important, color and other visuals are as well, said R.P. Higgins, president of Melbourne, Fla.-based Electric Picture Display Systems, which makes large simulation systems for military and civilian clients and has experienced the consumer-driven display technology push. Years ago, projector illumination was done by lamps, which tended to dim, so that after a few hundred hours a red buoy might appear yellow. The advent of LEDs fixed that problem, but at the cost of a decrease in brightness. Now laser phosphor illumination is being deployed. “It gives a steady light but probably at five or six times the brightness levels, and maybe more, of an LED,” Higgins said. The future is 8K Looking forward, resolution will continue to increase, he added. Today, the state-of-the-art for projection is 4K, or roughly 4000 pixels in the horizontal and 2000 vertical. On the horizon is 8K resolution, which has about four times as many pixels. The Oculus Rift, the HTC Vive and other head-mounted devices (HMDs) are a promising display trend but they have a drawback: Users cannot see what’s around them. Rockwell Collins, a Cedar Rapids, Iowa-based avionics and IT supplier, has a solution, said Craig Langhauser, product line manager in the simulations and training solutions group. The company mounts live action cameras on the goggle-like device, using these to capture surroundings. This is then blended with the virtual environment, creating a mixed reality display. Using cameras to capture nearby surroundings enables merged reality, allows trainees to see their own hands and body, and can improve training. Courtesy of Rockwell Collins. “Most importantly, the trainee sees themselves. They see their hands, their own feet, their own gear,” Langhauser said. “It gives a truer sense of immersion and connection to the training. Therefore, you get better training.” The visual representation does not have to be perfect, but it, along with everything else, must be good enough to convince users. Typically, this means that a combination of visuals, sounds and motion must be employed, with the fidelity of each varying by application, Langhauser said. Looking to the future, he cited the past. More than a decade ago, Rockwell Collins produced a custom system for the U.S. Army because consumer technology lacked the necessary performance. When that was no longer true, the cost fell from tens of thousands to a few hundred dollars. Something similar is happening to VR, AR and MR applications, according to Langhauser. For another use case, consider efforts by the DiSTI Corp. of Orlando, Fla. It has been investigating the use of see-through self-contained units, like Microsoft’s HoloLens, for mixed reality maintenance training. An important advantage of this approach is that it allows a collaborative work environment. Merged reality, which combines the real and synthetic, is being used for maintenance training, in part because it allows collaborative training. Courtesy of DiSTI. “Maintenance is done in teams, and so should be the training,” said Scott Ariotti, director of global marketing. With mixed reality, trainees can be presented with a synthetic environment that can contain real mockups that must be physically manipulated. What’s more, each trainee can get an individual view. However, current HMDs need to have higher resolution, a wider field of view, a faster refresh rate, be brighter and have an improved battery life, Ariotti said. Over time, these issues may be resolved. For that to happen, though, it will be because the fix aligns with a mass market use, like gaming or entertainment. Then the military will be able to take advantage of the advances, leading to changes in the training ecosystem. MetaVR Inc. of Brookline, Mass., has been making PC-based image generators, which transform a database into a virtual scene, for 20 years, said Philip Winston, lead software engineer. Traditionally these have run the multiple channels found in a large, domed simulator. The arrival of HMDs of adequate resolution could change that. An image generator is running in four displays — three that can be seen and a fourth within the JTAC’s (joint terminal attack controller’s) Ranger 47 binoculars/laser designator on the tripod. This shows training using a 4-meter 220-degree cylindrical display featuring a seamless high-fidelity image surface. Courtesy of MetaVR, Close Air Solutions and Battlespace Simulations. “Instead of one guy training, you have 50 guys and they all have HMDs,” Winston said, in discussing an expansion of the image generator audience and target. He added that MetaVR invests heavily in its software. Much of this involves tailoring the image generator so that it can take advantage of hardware improvements, such as the move from one to many processor cores. An example of this application expansion can be seen in a system developed by Quantum3D, a Milpitas, Calif.-based training and simulation provider. Jan Bjernfalk, vice president of products, recalled a request to add a medic to a helicopter evacuation trainer. The medic didn’t need the same degree of visual fidelity as a pilot, but what was presented to the medic did have to sync up with what others saw. Current head-mounted displays provided a good-enough representation, for much less than a separate domed display. A Navy Seahawk helicopter full-mission simulator. Such virtual reality applications include sight, sound and motion to create a realistic-seeming situation suitable for training. Courtesy of Quantum3D. “They’re virtually inside the helicopter. They can be looking at a virtual patient,” Bjernfalk said. “They need to be able to communicate with the rest of the crew and do their job.” Even if head-mounted and other simulation displays improve, the advances won’t be entirely without cost to the military. As resolution moves up, so too does the file size, boosting bandwidth and processing demands. When “good enough” is OK There also is greater data density to consider, said Vlad Argintaru, project engineering manager at Aero Simulation Inc. of Tampa, Fla. At one time the imagery resolution in a database was between one and 16 meters. Now synthetic environments are built from imagery resolution of as much as 30 centimeters and terrain elevation posts eight or less meters apart. Image generators for flight training must compute a scene 60 times a second. With today’s projector displays, this can mean doing so for four million pixels, Argintaru said. The quality of the rendering by the display is critical. “What is more important than a high pixel count is to have smear reduction techniques that render better-quality pixels,” Argintaru said. If HMDs can replace more expensive displays, then the cost savings can be substantial, said Dan Brockway, vice president of marketing at VT MÄK, a Cambridge, Mass.-based company that makes image generator software. Although options exist when constructing a VR, AR or MR representation, there is a downside when leveraging consumer technology. Military users will be familiar with game and entertainment imagery and may expect to get the same level of visual detail. But games are built on a business model where large sums can be spent optimizing the virtual world, something not practical — and often not required — for military projects, Brockway said. He noted that a simulator only needs to have good-enough fidelity to make it effective. Hence, there may be no need for game-like visual content. “It’s a challenge to build all that content, but the technology is there to do it,” Brockway said.