Tangible Music Interface

No strings attached. Actually, no buttons, no switches and no keys either. A musical instrument that relies solely on digital image processing, the Reactable offers interaction between the musician and the instrument via the smooth surface of the illuminated table.

The Reactable was exhibited at this year’s Vision Show in Boston, and though it ended its 2007 world tour stint with Icelandic singer Bjork, it was still certainly one of the most popular interactive displays at the 2008 show.

This experimental instrument, developed by researchers from the Pompeu Fabra University of Barcelona in Spain, uses cameras from Allied Vision Technologies.

People interacting with the synthesizer interface, producing and transforming sounds.

The key to operating this new style of synthesizer lies in the various Plexiglas objects that are placed on the illuminated table, and how they are applied and moved.

Each object fulfills different functions based on geometric shapes. For example, square shaped elements generate basic tones, while round objects act as sound filters, modulating these basic tones. Each object has a different symbol that determines the type of the basic tone and/or the filter; the spatial relationship of the objects to each other determines the extent to which one element affects another.

A special collection of symbols was conceived for the Reactable to meet the system requirements for easy and fast recognition.

The Reactable projects markings onto the surface of the table to make the instrument easier to operate. These not only confirm to the musician that the object has been recognized by the system, but also provide additional information regarding the status of the generated tone and its interaction with neighboring objects. This allows the artist to see the connections and a dynamic graphic presentation of the generated sound waves on the table. What’s more, the musician can change individual sound parameters by touching the projected information with his/her finger.

An Allied Vision Technologies digital camera monitors everything that takes place on the table from underneath the glass plate. The specially developed image processing system ReacTIVision analyzes the images and derives corresponding sound information from the position of the objects. This is then transferred to the speakers as an audio signal and graphically projected onto the tabletop.

“With a camera and a constant bandwidth, we can monitor the entire surface, regardless of how many objects are on the table. In addition, image processing determines not only the position of the objects but also their orientation (rotation) and can recognize human fingers on the tabletop," says Sergi Jordà, director of the project at the Pompeu Fabra University of Barcelona. "With this, the optical solution offers many more fine tuning possibilities and nuances than alternative technologies and places almost no limits on musical creativity,”

What was critical for implementation, however, were the image transfer and processing speeds, as well as the sensitivity of the camera for the environmental lighting conditions. The instrument must react very quickly to the commands of the musician, especially if it is meant to play in synchronicity with a band.

“With the digital FireWire cameras by Allied Vision Technologies, we have a high data rate with more than 60 frames per second available to us, which makes the short reaction time of the instrument possible,” added Jordà.

The stability and reliability of the system is also extremely important. According to the researchers at Pompeu Fabra, the AVT cameras are also impressive in terms of lifetime and stability. In order to lessen the sensitivity to environmental light, the AVT cameras work in the near infrared spectrum (NIR). A special LED light is used in conjunction with a filter on the camera for this.

The Reactable is not yet being mass produced, but the actual prototypes have proven that the system can also function reliably in a live concert.

For more information, visit: www.alliedvisiontec.com , or www.upf.edu