Musical performance is a multimodal experience, for performers and listeners alike. This paper reports on a pilot study which constitutes the first step toward a comprehensive approach to the experience of music as performed. We aim at bridging the gap between qualitative and quantitative approaches, by combining methods for data collection. The purpose is to build a data corpus containing multimodal measures linked to high-level subjective observations. This will allow for a systematic inclusion of the knowledge of music professionals in an analytic framework, which synthesizes methods across established research disciplines. We outline the methods we are currently developing for the creation of a multimodal data corpus dedicated to the analysis and exploration of instrumental music performance from the perspective of embodied music cognition. This will enable the study of the multiple facets of instrumental music performance in great detail, as well as lead to the development of music creation techniques that take advantage of the cross-modal relationships and higher-level qualities emerging from the analysis of this multi-layered, multimodal corpus. The results of the pilot project suggest that qualitative analysis through stimulated recall is an efficient method for generating higher-level understandings of musical performance. Furthermore, the results indicate several directions for further development, regarding observational movement analysis, and computational analysis of coarticulation, chunking, and movement qualities in musical performance. We argue that the development of methods for combining qualitative and quantitative data are required to fully understand expressive musical performance, especially in a broader scenario in which arts, humanities, and science are increasingly entangled. The future work in the project will therefore entail an increasingly multimodal analysis, aiming to become as holistic as is music in performance.

Instrument design is not just a matter of hardware, it also concerns strategies for software mapping of input to output data. I will in this paper report on how an augmented clarinet that I, together with a team at LTU began developing in 2015 has continued to develop over the past seven years. The focus will be on the development of artistic applications within the system. Performers frequently describe the resistance of their instrument as a manifestation of the challenges they encounter when playing. One can argue that the goal of a skilled performer is to get rid of resistance, but it is in fact a central part of the relationship between performer and instrument. Acquiring technique and skill seems to be a way for the performer not to overcome resistance but to learn how the instrument responds to force. Realizing the importance of resistance in the artistic process we need to ask ourselves; how can we use this knowledge when creating new instruments? Returning to video documentation of performances with two different mappings and through stimulated recall analysis I seek a deeper understanding of how software, mapping and performance practice interacts, forming the basis of the artistic expression.

Recent developments in artificial intelligence (AI) have been increasingly driven by technoscientific and corporate approaches that emphasise large-scale datasets and autonomous generation systems. In response, Human-Centred AI proposes an alternative framework foregrounding human agency, values, and creative control. This paper contributes to this discourse by examining collaborative performance with AI-augmented instruments through a practice-led experiment involving two musicians. In this way, we investigate how musical agency is distributed across humans and multiple AI systems. Conducted through a laboratory process leading up to a live performance, this project went through five phases: 1) introduction of interfaces, 2) curation of datasets, 3) training of neural audio synthesis models and applying corpus-based synthesis techniques, 4) working with the intelligent instruments in rehearsals and performance, and 5) analysing the outcomes. Through combining qualitative, phenomenologically grounded methods and practice-led artistic exploration, we identify emergent creative relationships between performers and AI-augmented instruments. By analysing the agency at play, we unpack how creative control is distributed between human and machine. Situating our work within the framework of professional collaborative performance, we address the lack of phenomenological research into intelligent instruments whilst contributing methodologies for accountable, artist-centred AI development in musical contexts.

This article presents a study that applies Laban’s Effort theory to detect a musician’s expressive intentions during performance. Laban Effort theory was chosen for its capacity to support the observation, description, and interpretation of expressive movement. The aim was to develop interactive musical systems that learn from the embodied expressivity musicians cultivate through practice. The study adopts a multidisciplinary approach, combining Laban Motion Analysis with interactive machine learning. Gesture data, audio, and video were recorded during a performance of Brahms’s Clarinet Sonata Op. 120 No. 2. The video was then analysed by an expert annotator observing Laban Efforts. The annotated video was then reviewed in collaboration with the performer to incorporate Effort Phrasing that reflects changes in intensity within Efforts. The annotations were then used as training data, together with motion data recorded during the performance, to train a regression model. The model was evaluated against expert annotations and through qualitative video analysis. Unlike the very linear notation of a Laban analyst, the model reflects the dynamism of human motion by capturing the almost humanly unobservable nuances in Effort variations. This points to the possibility of using machine learning models to reflect a performer’s expressive intentions in real-time.