An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized SoundShow others and affiliations
2023 (English)In: Computer music journal, ISSN 0148-9267, E-ISSN 1531-5169, Vol. 47, no 1, p. 64-84Article in journal (Refereed) Published
Abstract [en]
This article presents a custom system combining hardware and software that senses physiological signals of the performer's body resulting from muscle contraction and translates them to computer-synthesized sound. Our goal was to build upon the history of research in the field to develop a complete, integrated system that could be used by nonspecialist musicians. We describe the Embodied AudioVisual Interaction Electromyogram, an end-to-end system spanning wearable sensing on the musician's body, custom microcontroller-based biosignal acquisition hardware, machine learning–based gesture-to-sound mapping middleware, and software-based granular synthesis sound output. A novel hardware design digitizes the electromyogram signals from the muscle with minimal analog preprocessing and treats it in an audio signal-processing chain as a class-compliant audio and wireless MIDI interface. The mapping layer implements an interactive machine learning workflow in a reinforcement learning configuration and can map gesture features to auditory metadata in a multidimensional information space. The system adapts existing machine learning and synthesis modules to work with the hardware, resulting in an integrated, end-to-end system. We explore its potential as a digital musical instrument through a series of public presentations and concert performances by a range of musical practitioners.
Place, publisher, year, edition, pages
MIT Press Journals , 2023. Vol. 47, no 1, p. 64-84
National Category
Other Electrical Engineering, Electronic Engineering, Information Engineering
Research subject
Musical Performance
Identifiers
URN: urn:nbn:se:ltu:diva-108694DOI: 10.1162/comj_a_00672ISI: 001262355500008Scopus ID: 2-s2.0-85200370503OAI: oai:DiVA.org:ltu-108694DiVA, id: diva2:1891549
Funder
EU, European Research Council, (FP/2007-2013) / ERC Grant FP7-283771EU, Horizon 2020, grant agreement no. 789,825
Note
Godkänd;2024;Nivå 0;2024-08-22 (joosat);
Funder: French Agence Nationale de la Recherche (ANR-21-CE38-0018);
Full text license: CC BY
2024-08-222024-08-222025-10-21Bibliographically approved