Seeing Through Joyce’s Eyes

JOEL WILLIAMS, ASSOCIATE EDITOR
joel.williams@photonics.com

If you want to eat an elephant, conventional advice says to start with the first bite. To build a robot equipped with superhuman vision, ImmerVision says to start with the head.

Meet Joyce, a humanoid robot that has been under development for several months. Philosophically speaking, however, said Alessandro Gasparini, executive vice president of operations and chief commercial officer at ImmerVision, the project has been ongoing for 20 years.

A concept image of the Joyce robot. Courtesy of ImmerVision.

The project is unfinished, intentionally so, and was unveiled at the Embedded Vision Summit taking place virtually Sept. 15-25. Currently, the robot consists of a head equipped with 360° × 360° cameras, data-in-picture technology for synchronizing data, image processing algorithms, and open APIs.

The last detail is perhaps the most crucial to the project.

“The core of this is that basically the community can work together to achieve an industry goal — to bring the sense of perception to a robot,” said Alain Paquin, head of the Joyce project.

ImmerVision will be crowdsourcing technologies through community challenges to add enhanced capabilities and understandings to the robot. Joyce is the spiritual successor to a development kit that Gasparini had been working on for a couple of years to allow for greater collaboration within the computer vision community.

“There are a number of components on the market, there are a number of ways of integrating them, there are a number of algorithms, there are a number of everything. And we realize that ‘Okay, our expertise, it’s up to a certain level,’” Gasparini said.

Paquin used the anecdote of running a Roomba vacuum cleaner.

“The time that I can spend with a glass of wine in my hand watching [the vacuum cleaner] go around, it’s really impressive,” he said. “And at one point it frustrates me, because looking at it, I said, okay, they have all kinds of technology — they have this, they have that. Why is the robot bumping his head four times against the same spot in the wall?”

Paquin theorized that — noting the limits of a single company or a single group of people working on a project — a collaboration between various groups could yield a more intelligent robot, leading to the essential idea behind the Joyce project.

Gasparini said, “That’s the goal of Joyce. It’s really to open up a platform, a humanoid platform, a robot platform. That, instead of everything being developed by a closed system of engineers at ImmerVision, it’s to open it up to the whole community, to give a chance to smart people — whether they are a startup, a larger organization, a university — to bring some of their innovations in terms of technologies, sensing, to the system to enhance [Joyce's] abilities, to be able then to provide those features, like distance measurements, as a perception feature of the robot.”

In the near term, Joyce will be featured at industry events, where people will be able to log in with smart devices and see in 360° through the robot's eyes. The company has plans to bring Joyce to various locations, such as sporting events, concerts, road trips, the beach, helicopter tours, and even on skydiving trips. People will be able to log in for an immersive experience, to see and hear what Joyce experiences. Paquin said a virtual reality headset would make it especially immersive.

In the long term, as ImmerVision’s challenges arise, the robot will become more advanced. Features such as facial recognition, 3D perception, motion sensing, night vision, movement detection, and more will be added.

This innovative purpose is the inspiration for the community challenges. "Companies can then use the outputs of what they have built for Joyce to build [their own] devices,” Paquin said.

The level of integration required to build true robot perception will necessarily yield more advanced technologies. ImmerVision forecasts that the challenges will lead to improved technology for numerous applications, such as automotive, robotics, IoT, machine vision and learning, wearables, medical, drones, and more.

Eventually, the robot will be able to walk. But for now, the focus is on collaboration toward a singular goal — to bring computer vision beyond current possibilities.

“We would like, through this project, to push beyond the abilities that ImmerVision has or what our competencies are, but to be able to really ignite the ability to push the development of the device — a robot, a brain, a system that will be able to perceive like a human and beyond. And that, I believe, can only be done by a collectivity. It cannot be done by just one company,” Paquin said.