As a conductor Esa-Pekka Salonen marshals huge sonic forces. His own compositions, too, cover vast ground, from shadowy whispers to bone rattling explosions. But in an interview he is quiet and self-deprecating. Too quiet, I worry on a sunny afternoon on the rooftop terrace of DUMBO House in Brooklyn, where Salonen’s voice, inflected with the lilting rhythms of his native Finnish, has a hard time competing with the jaunty thump of the café’s playlist and the zippered drone of a helicopter streaking the sky.
As it happens, the nature of sound and the means to capture and creatively harness it are exactly what Salonen wants to talk about. In the classical music scene, innovative new technologies are transforming the standard symphony into a multisensory experience. With the Philharmonia Orchestra in London, where he has been the principal conductor since 2008, Salonen has already spearheaded the creation of symphonic virtual-reality experiences that plunge the listener into the heart of the ensemble.
“What usually goes unnoticed even by music lovers is the near-magical communication between musicians in a great orchestra,” Salonen says. “A very good orchestra can react in a fraction of a second to an impulse, a beat, an idea of a clarinet player mid-phrase. It has a neural network like a shoal of mackerel.” That communication becomes palpable to users who, donning a headset, experience a performance of, say, Stravinksy’s Rite of Spring from the perspective of a first-desk violist.
With the Philharmonia, the 61-year-old Salonen used VR to take “the experience of an orchestra to places where no orchestras ever go,” as he puts it. In 2020, he will become the music director of the San Francisco Symphony and seek ways to tap into the innovative spirit of Silicon Valley. He’s excited about the creative potential of augmented reality, in which technologies make it possible to superimpose virtual sensory information on top of a given natural environment. Take for example Bose’s openear headphones, sunglasses that have tiny speakers embedded in the frames that shoot sounds into the ear. Thanks to sensors and companion apps, the frames potentially know what a user is looking at and can deliver customized content. Other companies are making headsets that allow the wearer to see and hear virtual objects seemingly inserted into the 360-degree reality of their surroundings.
“There are some amazing possibilities,” Salonen says. He envisions a concert in which the live performance of the musicians is supplemented by a visual and dramatic virtual element. “The sound would be live,” he says, “and what you see is a mixture of reality and an added virtual layer.” If this sounds abstract, hard to picture, that’s part of the attraction for Salonen.
In his dual role as star conductor and composer, Salonen is one of the most visible advocates of new technologies in what is broadly defined as classical music, but he is far from alone. Across the field, composers are playing with the expressive potential of new technologies. In works by Rand Steiger, the sound whizzes about space thanks to real-time digital audio signal processing. The 2016 Pulitzer Prize finalist Ashley Fure takes sounds found in the environment and subjects them to sonogram analysis. The poignant, dramatic, or just plain humorous juxtaposition of live music and prerecorded samples continues to fascinate veteran composers like Eve Beglarian and Tan Dun.
Tod Machover, a composer at the Massachusetts Institute of Technology’s Media Lab, is also delving into augmented reality. With the Montreal Symphony Orchestra, he is experimenting with giving listeners open-ear frames that could someday allow them to essentially zoom in on a particular player at will. “Depending on where you look you would hear things emphasized that with the naked ear you can’t hear,” he said. Through subtle amplification, a phrase of the violas or the discreet plucked notes of the basses might take on more prominence. “I think of them as buried treasures.”
Machover’s team is also playing with the physical spaces in which music is heard to create a more visceral listening experience. A prototype of a suspended cube that moves, shakes, and undulates with music currently fits only two people. Its creators dream of designing a sensate chamber that holds 50. “We’re working on making the whole environment come alive,” Machover says.
Other composers are seizing on new technologies to turn acoustics—the way music behaves in a physical space—into an expressive device of its own. The Austrian composer Olga Neuwirth does this in a shimmering work, Le Encantadas, for which she captured the unique acoustics of a particular Venetian church. At a recent performance in Hamburg, the sonic properties of that church, complete with the ambient sound of lagoon water lapping at stone, took over the coolly gracious Elbphilharmonie hall.
“I wanted the room to transform itself like a bubble, expanding and caving in on itself,” Neuwirth says. “So that at times we seem to be in a small room with virtually no reverberance, at others in an almost infinite space.”
There is one other dimension that Salonen dreams of playing with: time itself. What if, he wonders, listeners could create their own timelines in a musical work, triggering particular sounds through gaze and movement? “Depending on where you are and what you’re looking at, the acoustic info would be different,” he says. “And you could return to a spot, but there would be consequences: The past would no longer be as it was.”
