In this chapter, we describe a series of studies related to our research on using gestural sonic objects in music analysis. These include developing a method for annotating the qualities of gestural sonic objects on multimodal recordings; ranking which features in a multimodal dataset are good predictors of basic qualities of gestural sonic objects using the Random Forests algorithm; and a supervised learning method for automated spotting designed to assist human annotators. The subject of our analyses is a performance of Fragmente2, a choreomusical composition based on the Japanese composer Makoto Shinohara’s solo piece for tenor recorder Fragmente (1968). To obtain the dataset, we carried out a multimodal recording of a full performance of the piece and obtained synchronised audio, video, motion, and electromyogram (EMG) data describing the body movements of the performers. We then added annotations on gestural sonic objects through dedicated qualitative analysis sessions. The task of annotating gestural sonic objects on the recordings of this performance has led to a meticulous examination of related theoretical concepts to establish a method applicable beyond this case study. This process of gestural sonic object annotation—like other qualitative approaches involving manual labelling of data—has proven to be very time-consuming. This motivated the exploration of data-driven, automated approaches to assist expert annotators.

Interwoven Sound Spaces is an interdisciplinary project which brought together telematic music performance, interactive textiles, interaction design, and artistic research. A team of researchers collaborated with two professional contemporary music ensembles based in Berlin, Germany, and Piteå, Sweden, and four composers, with the aim of creating a telematic distributed concert taking place simultaneously in two concert halls and online. Central to the project was the development of interactive textiles capable of sensing the musicians’ movements while playing acoustic instruments, and generating data the composers used in their works. Musicians, instruments, textiles, sounds, halls, and data formed a network of entities and agencies that was reconfigured for each piece, showing how networked music practice enables distinctive musicking techniques. We describe each part of the project and report on a research interview conducted with one of the composers for the purpose of analysing the creative approaches she adopted for composing her piece.

The Sophtar is a tabletop string instrument with an embedded system for digital signal processing, networking, and machine learning. It features a pressure-sensitive fretted neck, two sound boxes, and controlled feedback capabilities by means of bespoke interface elements. The design of the instrument is informed by my practice with hyperorgan interaction in networked music performance. I discuss the motivations behind the development of the instrument and describe its structure, interface elements, and the hyperorgan and sound synthesis interactions approaches it implements. Finally, I reflect on the affordances of the Sophtar and the differences and similarities with other instruments and outline future developments and uses.

This article presents a custom system combining hardware and software that senses physiological signals of the performer's body resulting from muscle contraction and translates them to computer-synthesized sound. Our goal was to build upon the history of research in the field to develop a complete, integrated system that could be used by nonspecialist musicians. We describe the Embodied AudioVisual Interaction Electromyogram, an end-to-end system spanning wearable sensing on the musician's body, custom microcontroller-based biosignal acquisition hardware, machine learning–based gesture-to-sound mapping middleware, and software-based granular synthesis sound output. A novel hardware design digitizes the electromyogram signals from the muscle with minimal analog preprocessing and treats it in an audio signal-processing chain as a class-compliant audio and wireless MIDI interface. The mapping layer implements an interactive machine learning workflow in a reinforcement learning configuration and can map gesture features to auditory metadata in a multidimensional information space. The system adapts existing machine learning and synthesis modules to work with the hardware, resulting in an integrated, end-to-end system. We explore its potential as a digital musical instrument through a series of public presentations and concert performances by a range of musical practitioners.

Sestina, av TCP/IP kvartett och Åsa Unander-Scharin är ett ca 40 minuter långt verk, där ny musik, som skapats genom en radikal omtolkning av Monteverdis partitur, omger varje sats av det ursprungliga verket. En central faktor är hur både en dansare och två musiker styr Studio Acusticums hyperorgel, och därigenom iscensätter en läsning av Monteverdi som stundtals går långt in i körverkets detaljer, och i andra delar söker ett fågelperspektiv och väver in körmusiken i en långt större form. 

TCP/INDETERMINATE PLACE QUARTET bildades 2021 och består av Federico Visi, Robert Ek, Mattias Petersson och Stefan Östersjö. Klarinettisten Robert Ek och tonsättaren Mattias Petersson är båda doktorander vid Musikhögskolan i Piteå och knutna till forskningsklustret GEMM (Gesture, Embodiment and Machines in Music) där Federico Visi gjort ett postdoktoralt forskningsprojekt. Klustret leds av Stefan Östersjö, professor i Musikalisk gestaltning och ämnesföreträdare vid Musikhögskolan. Kvartetten tilldelades ”Best Music Award” först vid NIME (New Interfaces for Musical Expression) i Shanghai och sedan vid “Audio Mostly” i Graz. Båda prisen delades ut 2021, med hänvisning till kvartettens banbrytande spel med distansstyrda orglar, som utvecklats inom ramen för det internationella projekt Global Hyperorgan. 

Purpose

Restoration of walking ability is a key goal to both stroke survivors and their therapists. However, the intensity and duration of rehabilitation available after stroke can be limited by service constraints, despite the potential for improvement which could reduce health service demands in the long run. The purpose of this paper is to present qualitative findings from a study that explored the acceptability of a haptic device aimed at improving walking as part of an extended intervention in stroke rehabilitation.

Design/methodology/approach

Pre-trial focus groups and post-trial interviews to assess the acceptability of Haptic Bracelets were undertaken with seven stroke survivors.

Findings

Five themes were identified as impacting on the acceptability of the Haptic Bracelet: potential for improving quality of life; relationships with technology; important features; concerns; response to trial and concentration. Participants were interested in the haptic bracelet and hoped it would provide them with more confidence making them: feel safer when walking; have greater ability to take bigger strides rather than little steps; a way to combat mistakes participants reported making due to tiredness and reduced pain in knees and hips.

Originality/value

Haptic Bracelets are an innovative development in the field of rhythmic cueing and stroke rehabilitation. The haptic bracelets also overcome problems encountered with established audio-based cueing, as their use is not affected by external environmental noise.

This chapter explores three systems for mapping embodied gesture, acquired with electromyography and motion sensing, to sound synthesis. A pilot study using granular synthesis is presented, followed by studies employing corpus-based concatenative synthesis, where small sound units are organized by derived timbral features. We use interactive machine learning in a mapping-by-demonstration paradigm to create regression models that map high-dimensional gestural data to timbral data without dimensionality reduction in three distinct workflows. First, by directly associating individual sound units and static poses (anchor points) in static regression. Second, in whole regression a sound tracing method leverages our intuitive associations between time-varying sound and embodied movement. Third, we extend interactive machine learning through the use of artificial agents and reinforcement learning in an assisted interactive machine learning workflow. We discuss the benefits of organizing the sound corpus using self-organizing maps to address corpus sparseness, and the potential of regression-based mapping at different points in a musical workflow: gesture design, sound design, and mapping design. These systems support expressive performance by creating gesture-timbre spaces that maximize sonic diversity while maintaining coherence, enabling reliable reproduction of target sounds as well as improvisatory exploration of a sonic corpus. They have been made available to the research community, and have been used by the authors in concert performance.

The Global Hyperorgan is an intercontinental, creative space for acoustic musicking. Existing pipe organs around the world are networked for real-time, geographicallydistant performance, with performers utilizing instruments and other input devices to collaborate musically through the voices of the pipes in each location. A pilot study was carried out in January 2021, connecting two large pipe organs in Piteå, Sweden, and Amsterdam, the Netherlands. A quartet of performers tested the Global Hyperorgan’s capacities for telematic musicking through a series of pieces. The concept of modularity is useful when considering the artistic challenges and possibilities of the Global Hyperorgan. We observe how the modular system utilized in the pilot study afforded multiple experiences of shared instrumentality from which new, synthetic voices emerge. As a long-term technological, artistic and social research project, the Global Hyperorgan offers a platform for exploring technology, agency, voice, and intersubjectivity in hyper-acoustic telematic musicking.