“Rise of the Machines,” the third “Terminator” sci-fi movie, features a future controlled by cold, calculating cyborgs that make their own decisions, including going back in time to exterminate the human resistance. While the evolution of machines using artificial intelligence (AI) and machine learning to design and build improved machines is still a long way off (much less the ability to time travel), smart factories — the highly productive phenomena at the center of the Industry 4.0/Industrial Internet of Things (IIoT) revolution — are growing and succeeding. Manufacturing in smart factories ideally involves robots using data-driven decision-making to automatically improve processes on the fly to produce everything from soup to aircraft more efficiently and cost-effectively. The video calling and voice- and eye-activation capabilities of HoloLens 2 enable projection of a hands-free virtual display with a 42° field of view, allowing trainees to view step-by-step how-to manuals or take video calls with technicians, while leaving the hands free for tools. Courtesy of Microsoft. Smart manufacturing — also called “lights-out manufacturing” or “dark factories” for their ability to operate without humans present and thus without lights or heat — uses all kinds of photonics technology, including lasers, optical sensors, HD cameras, photonic integrated circuits, touch displays, fiber optics, optical waveguides, and 3D machine vision, not to mention an abundance of associated lenses, prisms, and coatings. According to the 2019 “Smart Factories @ Scale” report by the Capgemini Research Institute in Paris, smart factories could add as much as $2.2 trillion to the worldwide economy by 2023 thanks to productivity gains, quality improvements, and customer services. Market research firm Kenneth Research in New York predicts the market for smart manufacturing will continue to grow at a compound annual growth rate of 15.75% through 2023. The largest manufacturing segment adopting smart technology is aerospace, driven by the increased need for advanced long-haul passenger aircraft. At the center of every advanced smart factory is big data, or rather the capability to connect, collect, and aggregate data from equipment, maintenance, production, operations, inventory, logistics, machine vision, and so on, and apply machine learning and AI to make decisions. The more data and history a robotic system gathers, the better the decisions it can make — in some cases, better and faster decisions than a human could make. But scaling up existing legacy manufacturing facilities to collect and act on all this data is a complex process with discrete steps, the first of which involves aggregating multiple data streams from a factory’s disparate systems (equipment status, operation and production data, logistics, and so on) into one source. This requires a new data architecture across all systems, which is an expensive and time-consuming obstacle for many manufacturers. The next step is installation of a machine learning platform that applies AI to all the data to predict machine failures and product defects, and generates recommendations for remediation, which can save costly downtime. Finally, a smart factory scales to its ultimate potential when the system uses its predictions to automatically execute the recommended remediation, such as changing the settings on part of an assembly line1. Even with such a process, each smart factory will have different priorities; no two will look exactly the same. While a brand-new smart factory is ideally designed from the ground up, existing industrial facilities require retrofitting. As existing equipment wears out, it must be replaced or retrofitted with new smart equipment capable of IIoT connectivity. Among the easiest smart solutions to implement in legacy factories to enable connectivity, facilitate design collaboration, and improve problem-solving, production, and training are augmented reality and mixed reality (AR/MR) headsets or helmets (Figure 1). Whereas virtual reality completely immerses the user in an experience via a vision-occluding headset, AR uses mobile devices such as phones, tablets, wearable headsets, or helmets to display data or objects over the environment of the user. Figure 1. Lightweight MR headsets such as the X2 MR smart glasses from rifle-scope specialist ThirdEye Gen in Princeton, N.J., have found use in industrial, health care, and business applications. Courtesy of ThirdEye Gen. “AR” and “MR” are often used loosely and interchangeably, but, by most definitions, MR technology is a deeper, more immersive level of user engagement. An AR platform (such as a cellphone camera, tablet, or headset) enables a user to view their real-world environment overlaid with computer-generated images, figures, or video, whereas an MR platform (so far, typically a headset) enables smart interaction between the real world and the virtual. For example, in an MR headset, a user may be able to use tactile gestures to rotate or position a holographic image of a component projected onto an actual piece of machinery in their field of view (Figure 2). Figure 2. Viewable 3D data can be manipulated with gestures initiated by hands, eyes, or voice, thanks to MR headsets such as the HoloLens 2. Courtesy of Microsoft. MR headsets Both AR and MR head-up displays are ideal for industrial smart factory applications, delivering operational feedback from equipment and systems to humans on the floor, reducing downtime and paperwork. The driver of MR wearables in smart factories is the ability to place a computer in the worker’s field of vision for increased production in its many guises. Market research firm Forrester Research predicts that approximately 14.4 million U.S. workers will wear AR or MR smart glasses by 2025, up from approximately half a million in 2019. The first smart glasses product, Google Glass, launched to consumers in 2013 with great hoopla but with less than resounding success, partly due to issues of privacy. Instead, Google found that Glass was a bigger hit in industrial and factory applications. More than two dozen companies are involved in developing smart headsets or helmets, including well-known multinational giants Canon, Epson, HTC, Intel, Qualcomm, and Sony. Like Google, many of them are focusing on industrial and business markets to supplement soft consumer sales. A sampling of smaller companies launching smart AR/MR headsets includes Magic Leap (Plantation, Fla.), MindMaze (Lausanne, Switzerland), Nreal (Beijing), ThirdEye Gen (Princeton, N.J.), and Vuzix (Rochester, N.Y.). Another company, DAQRI (Los Angeles), went out of business as of September 2019. At Mobile World Congress (MWC) Barcelona 2019, Microsoft made a splash with the introduction of the HoloLens 2, an immersive MR headset of lightweight carbon fiber with a transparent 3D holo- graphic MEMS display (Figure 3). The display has four cameras for head tracking, two IR cameras for eye tracking, a 52° field of view, 2048- × 1080-pixel resolution, and firmware that enables gesture manipulation of holograms — selecting, grabbing, sliding, and placing objects in the field of view (Figure 4). Alex Kipman, technical fellow for AI and mixed reality at Microsoft, demonstrated how the HoloLens 2 immediately identifies and authenticates the user from their iris upon donning the headset, and calibrates for the user’s unique hand size. The HoloLens 2 tracks the iris to follow where the user is looking to enable selecting and scrolling of a virtual image by hand. Not only that, the headset enables controlling the displayed image with only the eyes or voice — offering exciting possibilities for hands-free instruction and training. Figure 3. (top) The lightweight headset design of the HoloLens 2 moves the center of gravity farther back on the head for improved comfort over previous headsets. The see-through holographic waveguide lenses flip up when necessary (bottom). Four built-in video cameras track the user’s head position, while two IR cameras track eye motion. Courtesy of Microsoft. Many more AR/MR headset advancements followed in 2019, with a focus on business-use apps. In May, Google launched the Glass Enterprise Edition 2 AR head-up display featuring a video display that can handle computer vision and advanced machine learning capabilities, available as a software development kit (SDK) to enable creation of custom solutions for the workplace. Smart glasses supplier Vuzix and AR solutions provider Ubimax (Bremen, Germany) announced in October the launch of an everything-as-a-service (XaaS) subscription for AR glasses for inspection, remote support, and order picking. In December, Magic Leap announced its Enterprise Suite update to its Magic Leap One: Creator Edition headset, offering third-party apps ideal for collaboration, location-based tours, and 3D visualization for business and medicine. Figure 4. Projection of 3D holographic models can be shared and ‘handled’ virtually by remote designers to save on prototyping and time to market. Courtesy of Microsoft. Dozens more companies are partnering with smart headset makers to specialize in customized software platforms for specific AR/MR use cases in smart factories. Among them are software companies Atheer (Santa Clara, Calif.), Hexagon (Amsterdam), RE’FLECKT (Munich), Scope AR (San Francisco), and Upskill (Vienna, Va.). A smart factory scales to its ultimate potential. AR software developer PTC (Boston) has partnered with both Microsoft and Magic Leap to build AR solutions for clients such as ANSYS, Hewlett Packard, Howden, Rockwell Automation, and BAE Systems. PTC’s Vuforia Studio software platform helps companies build custom AR experiences and cross-platform capabilities for enterprise and IIoT applications. At MWC Barcelona 2019, engineering firm Howden (Glasgow, Scotland) demonstrated the use of Vuforia Studio and HoloLens 2 to create immersive displays of system statuses to reduce costly unplanned downtime in wastewater treatment plants (Figure 5). At BAE Systems in Binghamton, N.Y., PTC’s Vuforia Studio enabled custom interactive MR experiences on the HoloLens 2 that improved training speed by 30% to 40% and allowed step-by-step immersive guidance through complex assembly processes of electric propulsion systems found in buses. Figure 5. An engineer at Howden uses HoloLens 2 and PTC’s Vuforia Studio software to create an immersive MR overlay of equipment status on the physical system in a wastewater treatment plant. Courtesy of PTC. “Twenty years ago, we didn’t have personal computers on every bench. Ten years ago, we didn’t have 3D printers. Now I can’t imagine building without those,” said Shawn Atkinson, operations program manager at BAE Systems. “And I think this is what the HoloLens represents. It’s the next step in the evolution of high-tech manufacturing.” For the foreseeable future, the machines are rising only to help raise productivity. Reference 1. W. Sundblad (February 2019). The four levels of a smart factory evolution, www.forbes.com/sites/willemsundbladeurope/ 2019/02/05/the-four-levels-of-a-smart-factory-evolution/#37e6482c56f6. Smart Factories Smart factories are primarily about data — automating its collection and making it accessible and action-oriented, according to a recent Forbes article. Deloitte uses terms such as “interconnected landscape,” “self-optimized,” “predicative,” and “ongoing, flexible learning” to describe the evolving nature of operations and production in the smart factory. An on-site construction worker uses a smart helmet manufactured by ThirdEye Gen to plan plumbing. Courtesy of ThirdEye Gen. Honeywell incorporates Microsoft’s HoloLens 2 into training solutions. Courtesy of Microsoft. A thyssenkrupp Elevator service technician uses Microsoft’s HoloLens 2 for diagnostics and repair. Courtesy of Microsoft. A technician uses HoloLens 2 to follow virtual directions for assembling a generator. Courtesy of Microsoft.