AI, Sensor Fusion Turn the Key to Immersive Vehicle Imaging

JAKE SALTZMAN, NEWS EDITOR JAKE.SALTZMAN@PHOTONICS.COM

When it comes to car buying, one tool is more valuable than the rest: our eyes. For as long as we have bought and sold cars, we have used manual inspection to find the dents and dings that inform our decisions about vehicle purchases.

3D computer vision and machine learning company Fyusion is expanding our ability to view and evaluate vehicles, inside and out. The company puts visual data behind a vehicle’s every imperfection, in the form of 2D images that the company’s technology then converts to 3D.

With Fyusion’s technology, prospective sellers use their smartphone’s image capture capabilities to place virtual buyers on the lot and make spot determinations about a vehicle’s quality.

The heat map function is applied to an image of a vehicle exterior. Top-down and side views (inset) indicate the same damages in a different platform. Fyusion’s AI Lightfield Information Suite (ALIS) engine powers a sensor fusion mechanism that allows the company to process and deliver visual data captured via smartphone. Courtesy of Fyusion.

The aim is to deliver transparency about a vehicle’s condition. The imaging solution sifts through thousands of points of visual data to deliver information about everything from door dings to minor scratches.

Wholesale buyers are among those who reap the benefits. Dealerships, car rental agencies, courier services, and any other fleet operators often purchase cars in bulk. Fyusion allows auto buyers and sellers to make real-time analyses of a vehicle’s condition.

The ability to do so is especially useful in the wholesale market — specifically on the auction lot, where a car may be one of 100 or more up for sale, and visual differentiations may not be immediately apparent. The solution helps buyers who may not be on location to use their mobile device to see a car in three dimensions.

Sensor fusion via AI platform

The combination of a regular smartphone and Fyusion’s AI-Based Lightfield Information Suite (ALIS) is at the core of Fyusion’s technology.

“The best way to describe it is as a cocktail of algorithms,” said Radu Rusu, Fyusion’s CEO. “We brought them together for pure machine learning, computer vision, and 3D processing. And then we built a whole set of things on top of these that do machine learning in a 3D space.

“People like to use fancy words, like it’s a ‘neural core,’ but it’s really almost like a puzzle, where you put things together to create pipelines for processing and go from one specific input to one specific output.”

The ALIS platform allows a smartphone to perform at its peak potential as an imaging device. With its multiple camera streams, complete with gyroscopes and an array of other sensors, a smartphone is an impressive computing device.

Whereas it succeeds in capturing a volume of images, a smartphone on its own is essentially a data bank. ALIS brings a smart device’s sensors together with the data from the images that they collect.

The sensor fusion mechanism also processes and delivers visual data from other instruments, such as accelerometers and magnetometers.

Fyusion’s Inspect3D image analysis platform enables a top-down image view option. It also allows users to acquire 3D images simply by walking around the perimeter of a vehicle. The multidimensional approach for vehicle inspection provides a heat map that indicates areas of damage and the level of severity of each imperfection. Courtesy of Fyusion.

“With ALIS, we are taking data from all of those components and computing two different things,” Rusu said. “We are computing the optics — physical characteristics, in 3D — to understand the shape and appearance of what we are capturing. And then we are computing the camera position relative to the car, so that we can precisely determine the structure of the object, just by walking around.”

Capture, Engine, Viewer modules

The ALIS platform enables three distinct modules, two of which — Capture and Engine — allow users to create vehicle condition reports. Once an image is captured, Fyusion software performs a visual condition analysis that identifies imperfections that may not be visually apparent in 2D or even in 3D.

Fyusion’s damage analysis tool is called Inspect3D. Once a user captures images using the Capture module, the images enter the Engine module. Inspect3D then generates a condition report based on the 3D images.

The capability can differentiate between conditions such as dirt smudges and dents, for example. One functionality even produces a type of heat map for the user, to further spotlight individual parts or components that are in particularly pristine or poor condition.

The third module, Viewer, is aptly named: It offers direct support to the viewer. Once images have been captured, Fyusion deploys its own 3D file format: the “.fyuse” file. The format provides a way to share 3D images without requiring a lot of memory. The file type is fully interactive and compatible with multiple platforms.

Wholesale from a distance

Many roads await a car after it leaves the factory. Damages can occur even before a car becomes available for sale. When a car does go to market, rental agencies, dealerships, and commercial fleets may all emerge as potential buyers. Many cars ultimately end up at auction.

For Fyusion, wholesale car buying is the sweet spot in the market.

Fyusion CTO and co-founder Stefan Holzer said the rise in online car buying was noticeable pre-COVID-19. The pandemic further expedited the shift in that direction. With more prospective buyers turning to online buying options, condition reports based on 3D images are an attractive tool, especially for bulk buyers. Manual inspections take time and require travel. Performing bulk manual inspections is an even greater challenge due to labor and cost.

“On the one hand, we are allowing people to come and go to the auction and explore the wholesale world — to be able to go to the parking lots, actually go into the vehicle, and be able to see why a price is that price,” Rusu said. “On the other hand, we are allowing faster throughput. Moving metal is expensive and time-consuming. There is a physical footprint.”

Since 2018, Fyusion has collaborated with Manheim, a global provider of end-to-end wholesale vehicle solutions. The company offers nearly 6 million used vehicles each year at physical and digital auctions. It is the leading automotive wholesaler in North America.

As part of the collaboration, Fyusion delivers 3D images to the Manheim Express app. The app uses Fyusion’s 3D images to help wholesalers buy and sell whenever they want from wherever they are, and it enables dealers to remarket inventory without the need to be in the presence of the car.

The same Fyusion features that are available to Inspect3D users via the ALIS platform, such as visual tags, are available to users of the Manheim Express app.

According to the collaborators, sales through Manheim Express increased approximately 230% between 2020 and 2021.

In January, Fyusion was acquired by Cox Automotive, the company under which the Manheim brand operates. Fyusion and Manheim plan to continue to innovate and refine their offerings, which now include the ability to generate a wholesale listing in minutes based on 360°, moveable images of a car’s interior and exterior.

Automated condition reporting

The big picture at Fyusion involves a pair of solutions, Rusu said. Work continues on the first: to ensure that prospective buyers who want to view a car remotely can do so as well as, if not better than, if they were there in person.

The second solution involves automated condition reports. In other words, cars that build condition reports of themselves.

The same logic that allowed Fyusion’s smartphone image capture technology to take off over the last eight years now applies in the area of generating automated reports.

Smartphones and smartphone sensor and camera technology have advanced drastically since Fyusion was formed in 2014, Holzer said. Fyusion’s Capture, Engine, and Viewer modules transformed in concert with these advancements. In its early years, Fyusion adapted its technology. Rather than rely on the sensors that were available for integration into devices at the time, the company opted to use software, computer vision, and AI algorithms to do the work that existing sensor tech was unable to perform.

The sensor and camera technologies eventually caught up. The same Fyusion technology that would once overwhelm the performance capabilities of a smartphone now works seamlessly on the device — which may be equipped with multiple cameras and even lidar. This has led to new opportunities for the company.

Fyusion remains focused on sensor fusion. Now, smart cars, autonomous driving, and electric vehicles are all part of the company’s equation.

“If you think about these vehicles, they are really just a collection of sensors strapped to the body of a car. It becomes more about computing than about the actual driving,” Rusu said.

If a sensor is off by even a fraction because the body of the car is slightly deformed or scratched, the influence on the car’s ability to drive or act autonomously could be significant.

“Being able to have cars build condition reports of themselves as they drive on the highway — it’s a crazy, futuristic vision,” he said. “But from a technology perspective, what we are shooting for is complete dominance in solving this problem.”