Technology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience and usability. More than one modality can be provided to an agent for interaction with a system to accomplish this goal, which is referred to as a multimodal interaction (MI) system. The Internet of Things (IoT) and augmented reality (AR) are popular technologies that allow interaction systems to combine the real-world context of the agent and immersive AR content. However, although MI systems have been extensively studied, there are only several studies that reviewed MI systems that used IoT and AR. Therefore, this paper presents an in-depth review of studies that proposed various MI systems utilizing IoT and AR. A total of 23 studies were identified and analyzed through a rigorous systematic literature review protocol. The results of our analysis of MI system architectures, the relationship between system components, input/output interaction modalities, and open research challenges are presented and discussed to summarize the findings and identify future research and development avenues for researchers and MI developers.

Museums and cultural heritage institutions have an important role in presenting accurate information and sharing cultural knowledge, and new technologies are increasingly implemented. For the best results, the appropriateness of a specific technology must be evaluated for each context. Research has shown the need for participatory methods and local knowledge in Indigenous design contexts. We describe a case study where an Indigenous Sámi museum exhibition was augmented with interactive technology through multidisciplinary co-design work with museum experts, designers, and developers. The traditional clothing of the Sámi people was digitized by filming, and information related to it was presented as a touchscreen installation in a renewed exhibition. User tests including interactive tasks and interviews (n = 7) and a questionnaire (n = 27) were completed on-site. The installation was rated interesting and easy to use, while some users struggled to find all the features. Our study shows that a technically relatively simple digital installation can be easy-to-use but interesting for a standard museum visitor. Additionally, the work demonstrates how to build successful collaborations that highlight Indigenous cultural heritage. We discuss the implications of using technology to promote cultural heritage and identities.

Despite the high number of educational games released, only a few games have a strong story that is more than an excuse for players’ actions. Furthermore, even fewer story-based games utilise the affordances of augmented reality (AR) to concretise abstract concepts while engaging players.Based on our literature review, we were inspired to merge AR into a story-based educational mobile game for teaching fractions to elementary school students. The game Tales & Fractions was created through a two-phase multidisciplinary development process. In order to successfully integrate AR into a story-based educational game, we employed an adapted version of the Scrum agile software development method implemented by a multidisciplinary team of experts from computer science, pedagogy, design and arts. During the development process, we faced many issues that other story-based AR game developers could meet. We summarised the encountered issues with our solutions which could be useful for developers to avoid common pitfalls and to enrich the user engagement.

The rise in the aging population worldwide is already negatively impacting healthcare systems due to the lack of resources. It is envisioned that the development of novel Internet of Things (IoT)-enabled smart city healthcare systems may not only alleviate the stress on the current healthcare systems but may significantly improve the overall quality of life of the elderly. As more elderly homes are fitted with IoT, and intelligent healthcare becomes the norm, there is a need to develop innovative augmented reality (AR) based applications and services that make it easier for caregivers to interact with such systems and assist the elderly on a daily basis. This paper proposes, develops, and validates an AR and IoT-enabled healthcare system to be used by caregivers. The proposed system is based on a smart city IoT middleware platform and provides a standardized, intuitive and non-intrusive way to deliver elderly person's information to caregivers. We present our prototype, and our experimental results show the efficiency of our system in IoT object detection and relevant information retrieval tasks. The average execution time, including object detection, communicating with a server, and rendering the results in the application, takes on average between 767ms and 1,283ms.

Background: The older population needs solutions for independent living and reducing the burden on caregivers while maintaining the quality and dignity of life.

Objective: The aim of this study was to design, develop, and evaluate an older adult health care app that supports trained caregivers (ie, formal caregivers) and relatives (ie, informal caregivers). We aimed to identify the factors that affect user acceptance of interfaces depending on the user’s role.

Methods: We designed and developed an app with 3 user interfaces that enable remote sensing of an older adult’s daily activities and behaviors. We conducted user evaluations (N=25) with older adults and their formal and informal caregivers to obtain an overall impression of the health care monitoring app in terms of user experience and usability. In our design study, the participants had firsthand experience with our app, followed by a questionnaire and individual interview to express their opinions on the app. Through the interview, we also identified their views on each user interface and interaction modality to identify the relationship between the user’s role and their acceptance of a particular interface. The questionnaire answers were statistically analyzed, and we coded the interview answers based on keywords related to a participant’s experience, for example, ease of use and usefulness.

Results: We obtained overall positive results in the user evaluation of our app regarding key aspects such as efficiency, perspicuity, dependability, stimulation, and novelty, with an average between 1.74 (SD 1.02) and 2.18 (SD 0.93) on a scale of −3.0 to 3.0. The overall impression of our app was favorable, and we identified that “simple” and “intuitive” were the main factors affecting older adults’ and caregivers’ preference for the user interface and interaction modality. We also identified a positive user acceptance of the use of augmented reality by 91% (10/11) of the older adults to share information with their formal and informal caregivers.

Conclusions: To address the need for a study to evaluate the user experience and user acceptance by older adults as well as both formal and informal caregivers regarding the user interfaces with multimodal interaction in the context of health monitoring, we designed, developed, and conducted user evaluations with the target user groups. Our results through this design study show important implications for designing future health monitoring apps with multiple interaction modalities and intuitive user interfaces in the older adult health care domain.

As the impact of global warming on climate change becomes noticeable, the importance of energy efficiency for reducing greenhouse gas emissions grows immense. To this end, a platform, solution, and mobile apps are developed as part of the European Union’s Horizon 2020 research and innovation program to support energy optimization in residences. However, to ensure long-term energy optimization, it is crucial to keep users engaged with the apps. Since augmented reality (AR) and a virtual animal companion positively influenced user engagement, we designed an AR companion that represented the user’s residence states; thereby making the user aware of indoor information. We conducted user evaluations to determine the effect of the AR companion on user engagement and perceived usability in the context of energy management. We identified that the user interface (UI) with AR (ARUI) barely affected user engagement and perceived usability compared to the traditional UI without AR (TUI); however, we found that the ARUI positively affected one of the user engagement aspects. Our results show AR companion integration’s potential benefits and effects on energy management mobile apps. Furthermore, our findings provide insights into UI design elements for developers considering multiple interaction modalities with AR.