VR Opera Hits All the Right Notes

ROBIN RILEY, WEB EDITOR, robin.riley@photonics.com

This fall, visitors to venues across England will have the opportunity to experience two classic operas in virtual reality (VR).

"Magic Butterfly" is an immersive VR experience that reimagines scenes from "Madame Butterfly" and "The Magic Flute," featuring original recordings from the Welsh National Opera (WNO). It was created by VR/AR content production agency REWIND and the WNO, in collaboration with CAMERA, the University of Bath’s motion capture research center.

A screen shot from the 'Magic Butterfly' experience. Courtesy of Welsh National Opera.

The production combines the emotion and beauty of WNO soprano Karah Son’s voice with high-quality avatar imagery to envelope viewers in an interactive 3D performance.

David Massey, digital producer at WNO, told Photonics Media that, “Our vision for ‘Magic Butterfly’ was to create a piece of art that would stand or fall by its own artistic merits. We also wanted to create a theatrical experience that would naturally reach, engage and inspire both new and other audiences that might have no experience of opera.”

The team at REWIND shared Massey’s enthusiasm for this unusual VR project.

“From our first meeting, it was very clear that Welsh National Opera was excited about the same thing that we were — recreating traditional operas in ways that weren’t possible on stage,” said Greg Furber, REWIND’s director of VR.

CAMERA was commissioned to capture the soprano’s body movements only, but chose to be a bit more adventurous and capture Son’s face and the distinctive sleeves of her stage costume as well.

Usually a performer must wear head-mounted cameras to capture the face, Martin Parsons, CAMERA’s head of studio, told Photonics Media.

“Wearing such a headcam can inhibit the performance and adds extra overheads to the post processing,” Parsons said. “As REWIND had designed quite a stylized look, we only really needed to capture the broad movements. This was achieved by strategically placing a few relatively large markers on Karah’s face that could be seen by the motion capture cameras used for the body.”

To capture body and face motion data from Son while she performed the aria "Un Bel Di" from Puccini’s opera, the CAMERA team also needed to epitomize the fluttering movement of the sleeves on the avatar’s kimono.

It was not possible for Son to wear her stage costume, Parsons said, as this would have obstructed the cameras’ view of the optical markers.

Fortunately for the team, the stylized look of "Magic Butterfly" meant that they only needed to capture the overall movement of the sleeves. They achieved this by attaching light-reflecting markers to pieces of football goal netting worn by Son in place of sleeves, to mimic the movement of the fabric.

Soprano Karah Son, performing as Madame Butterfly for the Welsh National Opera. Courtesy of Jeremy Abrahams.
Furber was able to watch the markers being reconstructed in real time on the monitor. This allowed him and the REWIND team to see and judge the overall silhouette.

When Photonics Media asked Furber what the biggest challenges were in creating this VR experience, his immediate response was, “Definitely technical ones! We wanted to create a virtual reality experience in Unity that would run on Google’s Daydream platform. The level of visual detail we wanted in the experience was ambitious and so we took practices from our VFX background to push the visual quality, precision and realism of the project.

“Houdini is traditionally used for film, adverts, etc. to create super-real effects," Furber continued. "We employed it within the Madame Butterfly piece to create the complex butterfly movements. Houdini had never been used in a real-time mobile experience before, so a great deal of R&D was undertaken to successfully utilize the program.”

WNO, REWIND and CAMERA team leaders are enthusiastic about ‘Magic Butterfly’ and excited to share the results of their collaboration.

“One of the iconic images associated with Madame Butterfly is the form of the protagonist made from butterflies,” said Furber. “Using VR, it was finally possible to combine this key image with the performance to create a staging that is truly unique. The freedom that VR technology brings, coupled with the talent and presence of an amazing performer like Karah Son, created something very special.”

Soprano Karah Son (Welsh National Opera) wore a motion capture suit with mesh attached to mimic the movement of the sleeves of her kimono costume as she performed an aria from "Madame Butterfly." These data were used to drive an avatar for the "Magic Butterfly" virtual reality opera experience. Courtesy of CAMERA, University of Bath.

When asked what makes for a successful VR experience, Furber told Photonics Media: “To me, a successful VR experience makes the technology melt away. With ‘Magic Butterfly,’ we wanted to deliver a fully interactive and immersive experience suitable for people of all ages and tech abilities. A successful VR experience should be able to fully immerse the user and allow them to forget they’re wearing a headset. Having some control of the content is important too, as a sense of agency through interaction is what makes the user feel fully transported to our virtual worlds.”

WNO’s Massey said, “One of our aims was to see if audiences could emotionally respond to Karah’s performance as she performs as an avatar. I’d say that the reaction to ‘Magic Butterfly’ has been overwhelmingly positive, with many visitors taking a punt on VR and opera for the first time. This combination of music, drama and animation has been really effective, immersing audiences in the stories and moving some to tears.”

"The Magic Butterfly VR Experience" is free to visitors and will be on display at the following venues in England: Birmingham Hippodrome Square, Hurst Street, Birmingham, Oct. 31-Nov. 4; Albert Dock, Liverpool, Nov. 9-12; and the Victoria and Albert Museum, London, Jan. 19-21, 2018. Total running time is approximately eight minutes for the two VR operas.

For more details visit: https://www.wno.org.uk/event/magic-butterfly-vr-experience.