As Internet of Things (IoT) has developed, the number of connected entities has increased, allowing systems to interact with users and their environments in smart ways. For example, presence and environmental sensors enable the system to be aware of the user's state and environment, allowing it to provide helpful information for users. The connected entities include not only IoT devices for data acquisition and actuation but also interaction modalities for communication between users and systems. Multimodal interaction (MI) and augmented reality (AR) are enablers for enhanced user experience (UX) for IoT services. However, combining an IoT-enabled system with MI and AR without careful consideration may hinder the benefits of these technologies. Thus, understanding the technologies and target user group's characteristics depending on the application's context is essential. Design principles allow people—who are interested in UX and user interface (UI) development of IoT-enabled mobile AR applications—to gain knowledge about UX/UI design that considers both the technology and user aspects.This thesis aims to identify, propose, and validate design principles for mobile applications within IoT-enabled smart city domains, especially healthcare and energy management services. We identified the requirements and needs of MI and AR through a systematic literature review. We also found that studies of design principles for IoT-enabled mobile AR applications are limited. We designed, developed, and analyzed three IoT-enabled mobile AR applications along with their UX. From the state-of-the-art research, we compiled and categorized 26 existing design principles into seven categories. We derived and evaluated five new design principles based on the analysis of our developed applications. As a practical realization of the identified design principles, we provided examples of design principles through user interface mockups, which represent the re-designed interfaces of the applications. We expect that our findings will give insight into the UX/UI design of IoT-enabled mobile AR applications for researchers, educators, and practitioners interested in UX/UI development.

As the impact of global warming on climate change becomes noticeable, the importance of energy efficiency for reducing greenhouse gas emissions grows immense. To this end, a platform, solution, and mobile apps are developed as part of the European Union’s Horizon 2020 research and innovation program to support energy optimization in residences. However, to ensure long-term energy optimization, it is crucial to keep users engaged with the apps. Since augmented reality (AR) and a virtual animal companion positively influenced user engagement, we designed an AR companion that represented the user’s residence states; thereby making the user aware of indoor information. We conducted user evaluations to determine the effect of the AR companion on user engagement and perceived usability in the context of energy management. We identified that the user interface (UI) with AR (ARUI) barely affected user engagement and perceived usability compared to the traditional UI without AR (TUI); however, we found that the ARUI positively affected one of the user engagement aspects. Our results show AR companion integration’s potential benefits and effects on energy management mobile apps. Furthermore, our findings provide insights into UI design elements for developers considering multiple interaction modalities with AR.

Background: The older population needs solutions for independent living and reducing the burden on caregivers while maintaining the quality and dignity of life.

Objective: The aim of this study was to design, develop, and evaluate an older adult health care app that supports trained caregivers (ie, formal caregivers) and relatives (ie, informal caregivers). We aimed to identify the factors that affect user acceptance of interfaces depending on the user’s role.

Methods: We designed and developed an app with 3 user interfaces that enable remote sensing of an older adult’s daily activities and behaviors. We conducted user evaluations (N=25) with older adults and their formal and informal caregivers to obtain an overall impression of the health care monitoring app in terms of user experience and usability. In our design study, the participants had firsthand experience with our app, followed by a questionnaire and individual interview to express their opinions on the app. Through the interview, we also identified their views on each user interface and interaction modality to identify the relationship between the user’s role and their acceptance of a particular interface. The questionnaire answers were statistically analyzed, and we coded the interview answers based on keywords related to a participant’s experience, for example, ease of use and usefulness.

Results: We obtained overall positive results in the user evaluation of our app regarding key aspects such as efficiency, perspicuity, dependability, stimulation, and novelty, with an average between 1.74 (SD 1.02) and 2.18 (SD 0.93) on a scale of −3.0 to 3.0. The overall impression of our app was favorable, and we identified that “simple” and “intuitive” were the main factors affecting older adults’ and caregivers’ preference for the user interface and interaction modality. We also identified a positive user acceptance of the use of augmented reality by 91% (10/11) of the older adults to share information with their formal and informal caregivers.

Conclusions: To address the need for a study to evaluate the user experience and user acceptance by older adults as well as both formal and informal caregivers regarding the user interfaces with multimodal interaction in the context of health monitoring, we designed, developed, and conducted user evaluations with the target user groups. Our results through this design study show important implications for designing future health monitoring apps with multiple interaction modalities and intuitive user interfaces in the older adult health care domain.

Museums and cultural heritage institutions have an important role in presenting accurate information and sharing cultural knowledge, and new technologies are increasingly implemented. For the best results, the appropriateness of a specific technology must be evaluated for each context. Research has shown the need for participatory methods and local knowledge in Indigenous design contexts. We describe a case study where an Indigenous Sámi museum exhibition was augmented with interactive technology through multidisciplinary co-design work with museum experts, designers, and developers. The traditional clothing of the Sámi people was digitized by filming, and information related to it was presented as a touchscreen installation in a renewed exhibition. User tests including interactive tasks and interviews (n = 7) and a questionnaire (n = 27) were completed on-site. The installation was rated interesting and easy to use, while some users struggled to find all the features. Our study shows that a technically relatively simple digital installation can be easy-to-use but interesting for a standard museum visitor. Additionally, the work demonstrates how to build successful collaborations that highlight Indigenous cultural heritage. We discuss the implications of using technology to promote cultural heritage and identities.

The rise in the aging population worldwide is already negatively impacting healthcare systems due to the lack of resources. It is envisioned that the development of novel Internet of Things (IoT)-enabled smart city healthcare systems may not only alleviate the stress on the current healthcare systems but may significantly improve the overall quality of life of the elderly. As more elderly homes are fitted with IoT, and intelligent healthcare becomes the norm, there is a need to develop innovative augmented reality (AR) based applications and services that make it easier for caregivers to interact with such systems and assist the elderly on a daily basis. This paper proposes, develops, and validates an AR and IoT-enabled healthcare system to be used by caregivers. The proposed system is based on a smart city IoT middleware platform and provides a standardized, intuitive and non-intrusive way to deliver elderly person's information to caregivers. We present our prototype, and our experimental results show the efficiency of our system in IoT object detection and relevant information retrieval tasks. The average execution time, including object detection, communicating with a server, and rendering the results in the application, takes on average between 767ms and 1,283ms.

Technology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience and usability. More than one modality can be provided to an agent for interaction with a system to accomplish this goal, which is referred to as a multimodal interaction (MI) system. The Internet of Things (IoT) and augmented reality (AR) are popular technologies that allow interaction systems to combine the real-world context of the agent and immersive AR content. However, although MI systems have been extensively studied, there are only several studies that reviewed MI systems that used IoT and AR. Therefore, this paper presents an in-depth review of studies that proposed various MI systems utilizing IoT and AR. A total of 23 studies were identified and analyzed through a rigorous systematic literature review protocol. The results of our analysis of MI system architectures, the relationship between system components, input/output interaction modalities, and open research challenges are presented and discussed to summarize the findings and identify future research and development avenues for researchers and MI developers.

The development of technology has enabled diverse modalities that can be used by humans or machines to interact with computer systems. In particular, the Internet of Things (IoT) and Augmented Reality (AR) are explored in this report due to the new modalities offered by these two innovations which could be used to build multimodal interaction systems. Researchers have utilized multiple modalities in interaction systems for providing better usability. However, the employment of multiple modalities introduces some challenges that need to be considered in the development of multimodal interaction systems to achieve high usability. In order to identify a number of remaining challenges in the research area of multimodal interaction systems with IoT and AR, we analyzed a body of literature on multimodal interaction systems from the perspectives of system architecture, input and output modalities, data processing methodology and use cases. The identified challenges are regarding of (i) multidisciplinary knowledge, (ii) reusability, scalability and security of multimodal interaction system architecture, (iii) usability of multimodal interaction interface, (iv) adaptivity of multimodal interface design, (v) limitation of current technology, and (vi) advent of new modalities. We are expecting that the findings of this report and future research can be used to nurture the multimodal interaction system research area, which is still in its infancy.

Despite the high number of educational games released, only a few games have a strong story that is more than an excuse for players’ actions. Furthermore, even fewer story-based games utilise the affordances of augmented reality (AR) to concretise abstract concepts while engaging players.Based on our literature review, we were inspired to merge AR into a story-based educational mobile game for teaching fractions to elementary school students. The game Tales & Fractions was created through a two-phase multidisciplinary development process. In order to successfully integrate AR into a story-based educational game, we employed an adapted version of the Scrum agile software development method implemented by a multidisciplinary team of experts from computer science, pedagogy, design and arts. During the development process, we faced many issues that other story-based AR game developers could meet. We summarised the encountered issues with our solutions which could be useful for developers to avoid common pitfalls and to enrich the user engagement.