System disruptions
We are currently experiencing disruptions on the search portals due to high traffic. We are working to resolve the issue, you may temporarily encounter an error message.
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review
Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Computer Science.ORCID iD: 0000-0002-1536-5753
Department of Digital Media, Ajou University, Suwon 16499, Korea.ORCID iD: 0000-0001-5966-992x
Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Computer Science.ORCID iD: 0000-0002-8681-9572
2021 (English)In: Applied Sciences, E-ISSN 2076-3417, Vol. 11, no 4, article id 1738Article, review/survey (Refereed) Published
Abstract [en]

Technology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience and usability. More than one modality can be provided to an agent for interaction with a system to accomplish this goal, which is referred to as a multimodal interaction (MI) system. The Internet of Things (IoT) and augmented reality (AR) are popular technologies that allow interaction systems to combine the real-world context of the agent and immersive AR content. However, although MI systems have been extensively studied, there are only several studies that reviewed MI systems that used IoT and AR. Therefore, this paper presents an in-depth review of studies that proposed various MI systems utilizing IoT and AR. A total of 23 studies were identified and analyzed through a rigorous systematic literature review protocol. The results of our analysis of MI system architectures, the relationship between system components, input/output interaction modalities, and open research challenges are presented and discussed to summarize the findings and identify future research and development avenues for researchers and MI developers.

Place, publisher, year, edition, pages
MDPI, 2021. Vol. 11, no 4, article id 1738
Keywords [en]
multimodal interaction, interaction modalities, mixed/augmented reality, internet of things, systematic literature review, multimodal UI
National Category
Human Computer Interaction
Research subject
Pervasive Mobile Computing
Identifiers
URN: urn:nbn:se:ltu:diva-83646DOI: 10.3390/app11041738ISI: 000632121600001Scopus ID: 2-s2.0-85101409553OAI: oai:DiVA.org:ltu-83646DiVA, id: diva2:1544142
Funder
European Regional Development Fund (ERDF), 20201434, NYPS 20204318
Note

Validerad;2021;Nivå 2;2021-04-14 (alebob);

Finansiär: Ajou University (S-2020-G0001-00478)

Available from: 2021-04-14 Created: 2021-04-14 Last updated: 2024-10-18Bibliographically approved
In thesis
1. DesignWise: Design Principles for Smart Applications targeting Internet of Things Services
Open this publication in new window or tab >>DesignWise: Design Principles for Smart Applications targeting Internet of Things Services
2024 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

As Internet of Things (IoT) has developed, the number of connected entities has increased, allowing systems to interact with users and their environments in smart ways. For example, presence and environmental sensors enable the system to be aware of the user's state and environment, allowing it to provide helpful information for users. The connected entities include not only IoT devices for data acquisition and actuation but also interaction modalities for communication between users and systems. Multimodal interaction (MI) and augmented reality (AR) are enablers for enhanced user experience (UX) for IoT services. However, combining an IoT-enabled system with MI and AR without careful consideration may hinder the benefits of these technologies. Thus, understanding the technologies and target user group's characteristics depending on the application's context is essential. Design principles allow people—who are interested in UX and user interface (UI) development of IoT-enabled mobile AR applications—to gain knowledge about UX/UI design that considers both the technology and user aspects.This thesis aims to identify, propose, and validate design principles for mobile applications within IoT-enabled smart city domains, especially healthcare and energy management services. We identified the requirements and needs of MI and AR through a systematic literature review. We also found that studies of design principles for IoT-enabled mobile AR applications are limited. We designed, developed, and analyzed three IoT-enabled mobile AR applications along with their UX. From the state-of-the-art research, we compiled and categorized 26 existing design principles into seven categories. We derived and evaluated five new design principles based on the analysis of our developed applications. As a practical realization of the identified design principles, we provided examples of design principles through user interface mockups, which represent the re-designed interfaces of the applications. We expect that our findings will give insight into the UX/UI design of IoT-enabled mobile AR applications for researchers, educators, and practitioners interested in UX/UI development.

Place, publisher, year, edition, pages
Luleå: Luleå University of Technology, 2024
Series
Doctoral thesis / Luleå University of Technology 1 jan 1997 → …, ISSN 1402-1544
Keywords
Design principle, Internet of Things, augmented reality, multimodal interaction, user interface, user experience, human-computer interaction
National Category
Computer Sciences
Research subject
Pervasive Mobile Computing
Identifiers
urn:nbn:se:ltu:diva-110204 (URN)978-91-8048-684-2 (ISBN)978-91-8048-685-9 (ISBN)
Public defence
2025-01-24, A193, Luleå University of Technology, Skellefteå, 12:00 (English)
Opponent
Supervisors
Available from: 2024-10-18 Created: 2024-10-18 Last updated: 2024-12-12Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Kim, Joo ChanLaine, Teemu H.Åhlund, Christer

Search in DiVA

By author/editor
Kim, Joo ChanLaine, Teemu H.Åhlund, Christer
By organisation
Computer Science
In the same journal
Applied Sciences
Human Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 103 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf