Burning Meditations: Variational auto-encoders and timbre transfer in contemporary vocal and instrumental music

This talk presents the use of the RAVE algorithm from IRCAM in the composition and performance of two works: Burning Meditation for voice and AI, and Transfert for percussion ensemble, EMG and RAVE. Burning Meditation for baritone and electronics sets a poem by the Japanese beat poetess, Kazuko Shiaraishi, putting the human voice in dialogue with generative AI sound synthesis. The electronic part is based on an AI model of the performer’s own voice: the soloist thus is author of his own dataset. Interaction between the live voice of the singer and his own data model creates a dialogue with his double, deepening an exploration of his voice and reinforcing his sonic identity. Transfert is a piece for percussion, body sensing, and AI models, composed for three percussionists performing on metal, skin, and wood percussion. The instrumentalists play against three AI audio models created of them playing other, orchestral percussion instruments, using the RAVE variational autoencoder from IRCAM. The live percussion undergoes timbre transfer to the modelled sounds, resulting in a layering of acoustic percussion and an electronic, AI shadow of each instrumentalist. The corporeal gesture of each performer is captured by electrodes detecting the muscle electromyogram (EMG) signal from arm muscle tension and modulates the latent space of the modelled sounds, creating otherworldly sonic effects. The machine generated timbres will be produced in response to what is performed by the human percussionist entering into dialogue, one where the machine response is shaped by the physiology of the human performer. In both works, the audio models have been created in an ethical manner, modelling the very musicians who perform the works. This results in acoustic/electronic dialogues between instrumentalists and their sonic avatars.