publications | Florian Müller

2025

The Bitter Taste of Confidence: Exploring Audio-Visual Taste Modulation in Immersive Reality

Pooria Ghavamian, Jan Henri Beyer, Sophie Orth, Mia Johanna Nona Zech, Florian Müller, and Andrii Matviienko

In Proceedings of the 2025 ACM International Conference on Interactive Media Experiences, May 2025

PDF ACM

Extended Reality (XR) technologies present innovative ways to augment sensory experiences, including taste perception. In this study, we investigated how augmented reality (AR) visual filters and synchronized audio cues affect gustation through a controlled experiment with 18 participants. Our findings revealed unexpected crossmodal interactions: while pink visual filter typically associated with sweetness reduced perceived bitterness in isolation, it paradoxically enhanced bitterness perception when combined with sweet-associated audio cue. Furthermore, we observed an inverse correlation between participant confidence levels and their perception of taste intensities across multiple dimensions, highlighting confidence as an overlooked factor in sensory experience design. These findings inform the design of nuanced multisensory experiences in immersive media, where subtle crossmodal interactions significantly influence user perception.
Social MediARverse: Investigating Users’ Social Media Content Sharing and Consuming Intentions with Location-Based AR

Linda Hirsch, Florian Mueller, Mari Kruse, Andreas Butz, and Robin Welsch

Virtual Reality, Jul 2025

PDF Springer arxiv

Augmented Reality (AR) is evolving to become the next frontier in social media, merging physical and virtual reality into a living metaverse, a Social MediARverse. With this transition, we must understand how different contexts—public, semi-public, and private—affect user engagement with AR content. We address this gap in current research by conducting an online survey with 110 participants, showcasing 36 AR videos, and polling them about the content’s fit and appropriateness. Specifically, we manipulated these three spaces, two forms of dynamism (dynamic vs. static), and two dimensionalities (2D vs. 3D). Our findings reveal that dynamic AR content is generally more favorably received than static content. Additionally, users find sharing and engaging with AR content in private settings more comfortable than in others. By this, the study offers valuable insights for designing and implementing future Social MediARverses and guides industry and academia on content visualization and contextual considerations.
Don’t They Really Hear Us? A Design Space for Private Conversations in Social Virtual Reality

Josephus Jasper Limbago, Robin Welsch, Florian Müller, and Mario Di Francesco

IEEE Transactions on Visualization and Computer Graphics, Jul 2025

PDF IEEE

Seamless transition between public dialogue and private talks is essential in everyday conversations. Social Virtual Reality (VR) has revolutionized interpersonal communication by creating a sense of closeness over distance through virtual avatars. However, existing social VR platforms are not successful in providing safety and supporting private conversations, thereby hindering self-disclosure and limiting the potential for meaningful experiences. We approach this problem by exploring the factors affecting private conversations in social VR applications, including the usability of different interaction methods and the awareness with respect to the virtual world. We conduct both expert interviews and a controlled experiment with a social VR prototype we realized. We then leverage the outcomes of the two studies to establish a design space that considers diverse dimensions (including privacy levels, social awareness, and modalities), laying the groundwork for more intuitive and meaningful experiences of private conversation in social VR.
AR You on Track? Investigating Effects of Augmented Reality Anchoring on Dual-Task Performance While Walking

Julian Rasch, Matthias Wilhalm, Florian Müller, and Francesco Chiossi

In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems - CHI ’25, Jul 2025

PDF ACM arxiv

With the increasing spread of AR head-mounted displays suitable for everyday use, interaction with information becomes ubiquitous, even while walking. However, this requires constant shifts of our attention between walking and interacting with virtual information to fulfill both tasks adequately. Accordingly, we as a community need a thorough understanding of the mutual influences of walking and interacting with digital information to design safe yet effective interactions. Thus, we systematically investigate the effects of different AR anchors (hand, head, torso) and task difficulties on user experience and performance. We engage participants (n=26) in a dual-task paradigm involving a visual working memory task while walking. We assess the impact of dual-tasking on both virtual and walking performance, and subjective evaluations of mental and physical load. Our results show that head-anchored AR content least affected walking while allowing for fast and accurate virtual task interaction, while hand-anchored content increased reaction times and workload.
CreepyCoCreator? Investigating AI Representation Modes for 3D Object Co-Creation in Virtual Reality

Julian Rasch, Julia Töws, Teresa Hirzle, Florian Müller, and Martin Schmitz

In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems - CHI ’25, Jul 2025

PDF ACM arxiv

Generative AI in Virtual Reality offers the potential for collaborative object-building, yet challenges remain in aligning AI contributions with user expectations. In particular, users often struggle to understand and collaborate with AI when its actions are not transparently represented. This paper thus explores the co-creative object-building process through a Wizard-of-Oz study, focusing on how AI can effectively convey its intent to users during object customization in Virtual Reality. Inspired by human-to-human collaboration, we focus on three representation modes: the presence of an embodied avatar, whether the AI’s contributions are visualized immediately or incrementally, and whether the areas modified are highlighted in advance. The findings provide insights into how these factors affect user perception and interaction with object-generating AI tools in Virtual Reality as well as satisfaction and ownership of the created objects. The results offer design implications for co-creative world-building systems, aiming to foster more effective and satisfying collaborations between humans and AI in Virtual Reality.
A Vision for AI-Driven Adaptation of Dynamic AR Content to Users and Environments

Julian Rasch, Florian Müller, and Francesco Chiossi

Apr 2025

PDF arxiv

Augmented Reality (AR) is transforming the way we interact with virtual information in the physical world. By overlaying digital content in real-world environments, AR enables new forms of immersive and engaging experiences. However, existing AR systems often struggle to effectively manage the many interactive possibilities that AR presents. This vision paper speculates on AI-driven approaches for adaptive AR content placement, dynamically adjusting to user movement and environmental changes. By leveraging machine learning methods, such a system would intelligently manage content distribution between AR projections integrated into the external environment and fixed static content, enabling seamless UI layout and potentially reducing users’ cognitive load. By exploring the possibilities of AI-driven dynamic AR content placement, we aim to envision new opportunities for innovation and improvement in various industries, from urban navigation and workplace productivity to immersive learning and beyond. This paper outlines a vision for the development of more intuitive, engaging, and effective AI-powered AR experiences.
Through the Expert’s Eyes: Exploring Asynchronous Expert Perspectives and Gaze Visualizations in XR

Clara Sayffaerth, Annika Köhler, Julian Rasch, Albrecht Schmidt, and Florian Müller

Aug 2025

PDF arxiv

Transferring knowledge across generations is fundamental to human civilization, yet the challenge of passing on complex practical skills persists. Methods without a physically present instructor, such as videos, often fail to explain complex manual tasks, where spatial and social factors are critical. Technologies such as eXtended Reality and Artificial Intelligence hold the potential to retain expert knowledge and facilitate the creation of tailored, contextualized, and asynchronous explanations regardless of time and place. In contrast to videos, the learner’s perspective can be different from the recorded perspective in XR. This paper investigates the impact of asynchronous first- and third-person perspectives and gaze visualizations on efficiency, feeling of embodiment, and connectedness during manual tasks. The empirical results of our study (N=36) show that the first-person perspective is better in quantitative measures and preferred by users. We identify best practices for presenting preserved knowledge and provide guidelines for designing future systems.
From Pegs to Pixels: A Comparative Analysis of the Nine Hole Peg Test and a Digital Copy Drawing Test for Fine Motor Control Assessment

Dominik Schön, Thomas Kosch, Martin Schmitz, Sebastian Günther, Max Mühlhäuser, and Florian Müller

Proc. ACM Hum.-Comput. Interact., Sep 2025

PDF ACM data

User interaction with digital systems requires Fine Motor Control (FMC), especially if the interfaces are complex or require high fidelity and fine-grained interactions. Despite its importance, Fine Motor Control is often overlooked in interactive system design, partly because of its complex assessment. Measuring changes in fine motor abilities due to prolonged use or fatigue currently requires repeated manual testing. This paper analyzes the concept of using the digital mobile devices’ input behavior to assess the user’s Fine Motor Control. For this, we show that Fine Motor Control can be assessed for touch and stylus-based interaction with a digital mobile system. We conducted a user study, where participants performed a Nine Hole Peg Test and a predefined Copy Drawing Test before and after exercises that affect fine motor skills. Based on this data, we investigated how metrics such as pressure, velocity, and entropy for touch and stylus input can be used to predict Fine Motor Control.
Assessing Visualization and Interaction Techniques to Support Comparison Tasks in Virtual Reality

Henrike Weingärtner, Julian Rasch, Nils Rothamel, and Florian Müller

In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Apr 2025

PDF ACM

Desktop screens are effective for supporting comparison tasks, but as the scale increases to room-sized or larger structures, context is lost. Users are forced to focus on isolated details through panning, zooming, and scrolling, making it difficult to maintain an overview while exploring finer details. Virtual Reality (VR) potentially offers a solution to this problem by immersing users in 3D spaces and enabling more intuitive comparisons. While related work has proposed many solutions for visualizing and interacting for comparison tasks in desktop environments, knowledge regarding the efficacy of supporting such tasks in VR environments is still lacking. We investigated varying visualization and interaction techniques in a controlled experiment with 24 participants. Our findings provide valuable insights for designing VR systems that improve usability, reduce workload, and enhance performance in comparison tasks.
Investigating the Effects of Haptic Illusions in Collaborative Virtual Reality

Yannick Weiss, Julian Rasch, Jonas Fischer, and Florian Müller

IEEE Transactions on Visualization and Computer Graphics, Apr 2025

PDF IEEE

Our sense of touch plays a crucial role in physical collaboration, yet rendering realistic haptic feedback in collaborative extended reality (XR) remains a challenge. Co-located XR systems predominantly rely on prefabricated passive props that provide high-fidelity interaction but offer limited adaptability. Haptic Illusions (HIs), which leverage multisensory integration, have proven effective in expanding haptic experiences in single-user contexts. However, their role in XR collaboration has not been explored. To examine the applicability of HIs in multi-user scenarios, we conducted an experimental user study (N=30) investigating their effect on a collaborative object handover task in virtual reality. We manipulated visual shape and size individually and analyzed their impact on users’ performance, experience, and behavior. Results show that while participants adapted to the illusions by shifting sensory reliance and employing specific sensorimotor strategies, visuo-haptic mismatches reduced both performance and experience. Moreover, mismatched visualizations in asymmetric user roles negatively impacted performance. Drawing from these findings, we provide practical guidelines for incorporating HIs into collaborative XR, marking a first step toward richer haptic interactions in shared virtual spaces.
One Does Not Simply Meme Alone: Evaluating Co-Creativity Between LLMs and Humans in the Generation of Humor

Zhikun Wu, Thomas Weber, and Florian Müller

In Proceedings of the 30th International Conference on Intelligent User Interfaces, Mar 2025

PDF ACM arxiv

Collaboration has been shown to enhance creativity, leading to more innovative and effective outcomes. While previous research has explored the abilities of Large Language Models (LLMs) to serve as co-creative partners in tasks like writing poetry or creating narratives, the collaborative potential of LLMs in humor-rich and culturally nuanced domains remains an open question. To address this gap, we conducted a user study to explore the potential of LLMs in co-creating memes—a humor-driven and culturally specific form of creative expression. We conducted a user study with three groups of 50 participants each: a human-only group creating memes without AI assistance, a human-AI collaboration group interacting with a state-of-the-art LLM model, and an AI-only group where the LLM autonomously generated memes. We assessed the quality of the generated memes through crowdsourcing, with each meme rated on creativity, humor, and shareability. Our results showed that LLM assistance increased the number of ideas generated and reduced the effort participants felt. However, it did not improve the quality of the memes when humans were collaborated with LLM. Interestingly, memes created entirely by AI performed better than both human-only and human-AI collaborative memes in all areas on average. However, when looking at the top-performing memes, human-created ones were better in humor, while human-AI collaborations stood out in creativity and shareability. These findings highlight the complexities of human-AI collaboration in creative tasks. While AI can boost productivity and create content that appeals to a broad audience, human creativity remains crucial for content that connects on a deeper level.
Pull Requests From The Classroom: Co-Developing Curriculum And Code

Dennis Zyska, Ilia Kuznetsov, Florian Müller, and Iryna Gurevych

In Proceedings of Mensch Und Computer 2025 - MuC ’25, Aug 2025

PDF ACM arxiv

Educational technologies often misalign with instructors’ pedagogical goals, forcing adaptations that compromise teaching efficacy. In this paper, we present a case study on the co-development of curriculum and technology in the context of a university course on scientific writing. Specifically, we examine how a custom-built peer feedback system was iteratively developed alongside the course to support annotation, feedback exchange, and revision. Results show that while co-development fostered stronger alignment between software features and course goals, it also exposed usability limitations and infrastructure-related frustrations, emphasizing the need for closer coordination between teaching and technical teams.

2024

Embracer: A Wearable Encountered-Type Haptic Controller for 3 DoF Input and Feedback

Dennis Dietz, Steeven Villa, Moritz Ziarko, Michael Bonfert, Florian Müller, and Andreas Butz

In Proceedings of the 2024 ACM International Symposium on Wearable Computers, Oct 2024

PDF ACM

The lack of haptic sensations beyond very simple vibration feedback diminishes the feeling of presence in Virtual Reality. Research suggested various approaches to deliver haptic sensations to the user’s palm. However, these approaches are typically limited in the number of actuation directions and only focus on enhancing the system’s output, ignoring haptic input. We present Embracer, a wrist-mounted encountered-type haptic controller that addresses these gaps by rendering forces along three axes through a sphere-shaped end effector within the user’s palm. Using modified servo motors, we sense user-performed manipulations of the end effector as an input modality. In this paper, we contribute the design and implementation of Embracer together with a preliminary technical evaluation. By providing a more comprehensive haptic feedback system, Embracer enhances the realism and immersion of haptic feedback and user control.
Decide Yourself or Delegate - User Preferences Regarding the Autonomy of Personal Privacy Assistants in Private IoT-Equipped Environments

Karola Marky, Alina Stöver, Sarah Prange, Kira Bleck, Paul Gerber, Verena Zimmermann, Florian Müller, Florian Alt, and Max Mühlhäuser

In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, May 2024

PDF ACM Talk

Personalized privacy assistants (PPAs) communicate privacy-related decisions of their users to Internet of Things (IoT) devices. There are different ways to implement PPAs by varying the degree of autonomy or decision model. This paper investigates user perceptions of PPA autonomy models and privacy profiles – archetypes of individual privacy needs – as a basis for PPA decisions in private environments (e.g., a friend’s home). We first explore how privacy profiles can be assigned to users and propose an assignment method. Next, we investigate user perceptions in 18 usage scenarios with varying contexts, data types and number of decisions in a study with 1126 participants. We found considerable differences between the profiles in settings with few decisions. If the number of decisions gets high (> 1/h), participants exclusively preferred fully autonomous PPAs. Finally, we discuss implications and recommendations for designing scalable PPAs that serve as privacy interfaces for future IoT devices.
Just Undo It: Exploring Undo Mechanics in Multi-User Virtual Reality

Julian Rasch, Florian Perzl, Yannick Weiss, and Florian Müller

In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Mar 2024

PDF ACM arxiv Video Talk

With the proliferation of VR and a metaverse on the horizon, many multi-user activities are migrating to the VR world, calling for effective collaboration support. As one key feature, traditional collaborative systems provide users with undo mechanics to reverse errors and other unwanted changes. While undo has been extensively researched in this domain and is now considered industry standard, it is strikingly absent for VR systems in research and industry. This work addresses this research gap by exploring different undo techniques for basic object manipulation in different collaboration modes in VR. We conducted a study involving 32 participants organized in teams of two. Here, we studied users’ performance and preferences in a tower stacking task, varying the available undo techniques and their mode of collaboration. The results suggest that users desire and use undo in VR and that the choice of the undo technique impacts users’ performance and social connection.
“Tele” Me More: Using Telepresence Charades to Connect Strangers and Exhibits in Different Museums

Clara Sayffaerth, Julian Rasch, and Florian Müller

In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, May 2024

PDF ACM Video Talk

The museum is changing from a place of passive consumption to a place of interactive experiences, opening up new ways of engaging with exhibits and others. As a promising direction, this paper explores the potential of telepresence stations in the museum context to enhance social connectedness among visitors over distance. Emphasizing the significance of social exchange, our research focuses on studying telepresence to foster interactions between strangers, share knowledge, and promote social connectedness. To do so, we first observe exhibitions and then interview individual visitors of a technical museum about their experiences and needs. Based on the results, we design appropriate voiceless and touchless communication channels and test them in a study. The findings of our in-situ user study with 24 visitors unfamiliar with each other in the museum provide insights into behaviors and perceptions, contributing valuable knowledge on seamlessly integrating telepresence technology in exhibitions, with a focus on enhancing learning, social connections, and the museum experience in general.
Decoding Fatigue: Analyzing Offline Handwriting with Machine Learning to Detect Perceived Exhaustion

Dominik Schoen, Thomas Kosch, Till Becker, Godfred Antwi-Boasiako, Merret Jung, Ana Laura Chioca Vieira, Max Mühlhäuser, and Florian Müller

In Proceedings of the International Conference on Mobile and Ubiquitous Multimedia, Dec 2024

PDF ACM

The quality and readability of an individual’s handwriting and drawing can be influenced by various factors, including their level of physical exertion. This enables us to explore the quantification of exertion by observing an individual’s handwriting. To test this hypothesis, we collected data from 17 participants, building a database of handwriting and drawing samples and their corresponding Borg 10 exertion ratings at the time of drawing. In this paper, we investigate using machine learning techniques to estimate perceived exertion before, during, and after physical activity based on handwriting and drawings. We apply a regression model to compare different drawing tasks and demonstrate that perceived exertion can be predicted using simple line drawings. However, more complex sketches and handwriting demand further research. Our findings suggest that interactive systems could use handwriting and drawing to intervene when users experience excessive discomfort.
Touch It Like It’s Hot: A Thermal Feedback Enabled Encountered-type Haptic Display for Virtual Reality

Steeven Villa, Kenji Ishihara, Moritz Ziarko, Sebastian Günther, and Florian Müller

In IEEE International Symposium on Mixed and Augmented Reality (ISMAR’24) 2024, Dec 2024

PDF IEEE Talk

In recent years, the community has presented various novel solutions to address the lack of haptic feedback in virtual reality experiences. Yet, it remains a major challenge for Virtual Reality applications. Encountered-type Haptic Displays (ETHDs) have emerged as a promising alternative to enable haptic feedback in VR without requiring the user to wear any device while allowing for sensorily rich experiences such as texture, kinaesthetic feedback, and even ultrasonic tactile feedback. Nevertheless, as important as thermal feedback is for daily life interactions, such as assessing the temperature of a mug or knowing if the microwave is on, thermal feedback in ETHD has remained largely unexplored. In this paper, we present a novel ETHD that provides thermal feedback and explore its potential in VR. We describe the design of our ETHD, and we report the results of a user study that compares different thermal feedback settings in VR. Our results show that thermal feedback can significantly enhance the user immersion and haptic experience in VR, and we discuss the implications of our findings for the design of ETHD and VR experiences.
Exploring Redirection and Shifting Techniques to Mask Hand Movements from Shoulder-Surfing Attacks during PIN Authentication in Virtual Reality

Yannick Weiss, Steeven Villa, Jesse W Grootjen, Matthias Hoppe, Yasin Kale, and Florian Müller

Proc. ACM Hum.-Comput. Interact., Sep 2024

PDF ACM

The proliferation of mobile Virtual Reality (VR) headsets shifts our interaction with virtual worlds beyond our living rooms into shared spaces. Consequently, we are entrusting more and more personal data to these devices, calling for strong security measures and authentication. However, the standard authentication method of such devices - entering PINs via virtual keyboards - is vulnerable to shoulder-surfing, as movements to enter keys can be monitored by an unnoticed observer. To address this, we evaluated masking techniques to obscure VR users’ input during PIN authentication by diverting their hand movements. Through two experimental studies, we demonstrate that these methods increase users’ security against shoulder-surfing attacks from observers without excessively impacting their experience and performance. With these discoveries, we aim to enhance the security of future VR authentication without disrupting the virtual experience or necessitating additional hardware or training of users.

2023

In Sync: Exploring Synchronization to Increase Trust Between Humans and Non-humanoid Robots

Wieslaw Bartkowski, Andrzej Nowak, Filip Ignacy Czajkowski, Albrecht Schmidt, and Florian Müller

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

PDF ACM arxiv Video Talk

When we go for a walk with friends, we can observe an interesting effect: From step lengths to arm movements - our movements unconsciously align; they synchronize. Prior research found that this synchronization is a crucial aspect of human relations that strengthens social cohesion and trust. Generalizing from these findings in synchronization theory, we propose a dynamical approach that can be applied in the design of non-humanoid robots to increase trust. We contribute the results of a controlled experiment with 51 participants exploring our concept in a between-subjects design. For this, we built a prototype of a simple non-humanoid robot that can bend to follow human movements and vary the movement synchronization patterns. We found that synchronized movements lead to significantly higher ratings in an established questionnaire on trust between people and automation but did not influence the willingness to spend money in a trust game.
Digital Modeling for Everyone: Exploring How Novices Approach Voice-Based 3D Modeling

Giuseppe Desolda, Andrea Esposito, Florian Müller, and Sebastian Feger

In Human-Computer Interaction – INTERACT 2023, Apr 2023

PDF Springer

Manufacturing tools like 3D printers have become accessible to the wider society, making the promise of digital fabrication for everyone seemingly reachable. While the actual manufacturing process is largely automated today, users still require knowledge of complex design applications to produce ready-designed objects and adapt them to their needs or design new objects from scratch. To lower the barrier to the design and customization of personalized 3D models, we explored novice mental models in voice-based 3D modeling by conducting a high-fidelity Wizard of Oz study with 22 participants. We performed a thematic analysis of the collected data to understand how the mental model of novices translates into voice-based 3D modeling. We conclude with design implications for voice assistants. For example, they have to: deal with vague, incomplete and wrong commands; provide a set of straightforward commands to shape simple and composite objects; and offer different strategies to select 3D objects.
Understanding Stationary and Moving Direct Skin Vibrotactile Stimulation on the Palm

Hesham Elsayed, Martin Weigel, Florian Müller, George Ibrahim, Jan Gugenheimer, Martin Schmitz, Sebastian Günther, and Max Mühlhäuser

Feb 2023

PDF arxiv

Palm-based tactile displays have the potential to evolve from single motor interfaces (e.g., smartphones) to high-resolution tactile displays (e.g., back-of-device haptic interfaces) enabling richer multi-modal experiences with more information. However, we lack a systematic understanding of vibrotactile perception on the palm and the influence of various factors on the core design decisions of tactile displays (number of actuators, resolution, and intensity). In a first experiment (N=16), we investigated the effect of these factors on the users’ ability to localize stationary sensations. In a second experiment (N=20), we explored the influence of resolution on recognition rate for moving tactile sensations.Findings show that for stationary sensations a 9 actuator display offers a good trade-off and a \3}times3 resolution can be accurately localized. For moving sensations, a \2}times4 resolution led to the highest recognition accuracy, while \5}times10 enables higher resolution output with a reasonable accuracy.
ThermoFeet: Assessing On-Foot Thermal Stimuli for Directional Cues

Bettina Eska, Jeff-Owens Iyalekhue, Sebastian Günther, Jasmin Niess, and Florian Müller

In Proceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia, Dec 2023

PDF ACM

Thermal feedback has been studied for navigation purposes with directional cues and a variety of other use cases. Yet, to date, systems providing thermal feedback were primarily designed for the upper body, targeting hands and arms in particular. As these parts are often occupied with other tasks, there is a need to extend the design space of thermal feedback to other body parts. To close this gap, we assess thermal feedback on the user’s feet. This research explores if creating stimuli representing any direction on a circle with only four actuators is possible. To evaluate this concept, we conducted a user study asking the participants to indicate the perceived direction after getting a hot or cold stimulus by direct actuation using one actuator or phantom actuation using two actuators. The results indicate that the detection accuracy was higher for cold signals. In addition, the results showed higher recognition for stimuli linked to actuator distribution than phantom sensation due to spatial summation.
My Heart Will Go On: Implicitly Increasing Social Connectedness by Visualizing Asynchronous Players’ Heartbeats in VR Games

Linda Hirsch, Florian Müller, Francesco Chiossi, Theodor Benga, and Andreas Martin Butz

Proceedings of the ACM on Human-Computer Interaction, Oct 2023

PDF ACM Talk

Social games benefit from social connectedness between players because it improves the gaming experience and increases enjoyment. In virtual reality (VR), various approaches, such as avatars, are developed for multi-player games to increase social connectedness. However, these approaches are lacking in single-player games. To increase social connectedness in such games, our work explores the visualization of physiological data from asynchronous players, i.e., electrocardiogram (ECG). We identified two visualization dimensions, the number of players, and the visualization style, after a design workshop with experts (N=4) and explored them in a single-user virtual escape room game. We spatially and temporally integrated the visualizations and compared two times two visualizations against a baseline condition without visualization in a within-subject lab study (N=34). All but one visualization significantly increased participants’ feelings of social connectedness. Heart icons triggered the strongest feeling of connectedness, understanding, and perceived support in playing the game.
When XR and AI Meet - A Scoping Review on Extended Reality and Artificial Intelligence

Teresa Hirzle, Florian Müller, Fiona Draxler, Martin Schmitz, Pascal Knierim, and Kasper Hornbæk

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

PDF ACM Talk

Research on Extended Reality (XR) and Artificial Intelligence (AI) is booming, which has led to an emerging body of literature in their intersection. However, the main topics in this intersection are unclear, as are the benefits of combining XR and AI. This paper presents a scoping review that highlights how XR is applied in AI research and vice versa. We screened 2619 publications from 203 international venues published between 2017 and 2021, followed by an in-depth review of 311 papers. Based on our review, we identify five main topics at the intersection of XR and AI, showing how research at the intersection can benefit each other. Furthermore, we present a list of commonly used datasets, software, libraries, and models to help researchers interested in this intersection. Finally, we present 13 research opportunities and recommendations for future work in XR and AI research.
Location-Aware Virtual Reality for Situational Awareness On the Road

Jingyi Li, Alexandra Mayer, Florian Müller, Andrii Matviienko, and Andreas Butz

In Proceedings of the 2023 ACM Symposium on Spatial User Interaction, Oct 2023

PDF ACM

When future passengers are immersed in Virtual Reality (VR), the resulting disconnection from the physical world may degrade their situational awareness on the road. We propose incorporating real-world cues into virtual experiences when passing specific locations to address this. We designed two visualizations using points of interest (POIs), street names alone or combined with live street views. We compared them to two baselines, persistently displaying live cues (Always Live) or no cues (Always VR). In a field study (N=17), participants estimated their locations while exposed to VR entertainment during car rides. The results show that adding environmental cues inevitably degrades VR presence compared to Always VR. However, POI-triggered Text&Live preserves VR presence better than Always Live and attracts user attention to the road more than POI-triggered Text. We discuss situational awareness challenges for using mobile VR on the road and potential incorporation strategies across transport contexts.
TicTacToes: Assessing Toe Movements as an Input Modality

Florian Müller, Daniel Schmitt, Andrii Matviienko, Dominik Schön, Sebastian Günther, Thomas Kosch, and Martin Schmitz

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

PDF ACM arxiv Video Talk

From carrying grocery bags to holding onto handles on the bus, there are a variety of situations where one or both hands are busy, hindering the vision of ubiquitous interaction with technology. Voice commands, as a popular hands-free alternative, struggle with ambient noise and privacy issues. As an alternative approach, research explored movements of various body parts (e.g., head, arms) as input modalities, with foot-based techniques proving particularly suitable for hands-free interaction. Whereas previous research only considered the movement of the foot as a whole, in this work, we argue that our toes offer further degrees of freedom that can be leveraged for interaction. To explore the viability of toe-based interaction, we contribute the results of a controlled experiment with 18 participants assessing the impact of five factors on the accuracy, efficiency and user experience of such interfaces. Based on the findings, we provide design recommendations for future toe-based interfaces.
UndoPort: Exploring the Influence of Undo-Actions for Locomotion in Virtual Reality on the Efficiency, Spatial Understanding and User Experience

Florian Müller, Arantxa Ye, Dominik Schön, and Julian Rasch

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

PDF ACM arxiv Video Talk

When we get lost in Virtual Reality (VR) or want to return to a previous location, we use the same methods of locomotion for the way back as for the way forward. This is time-consuming and requires additional physical orientation changes, increasing the risk of getting tangled in the headsets’ cables. In this paper, we propose the use of undo actions to revert locomotion steps in VR. We explore eight different variations of undo actions as extensions of point&teleport, based on the possibility to undo position and orientation changes together with two different visualizations of the undo step (discrete and continuous). We contribute the results of a controlled experiment with 24 participants investigating the efficiency and orientation of the undo techniques in a radial maze task. We found that the combination of position and orientation undo together with a discrete visualization resulted in the highest efficiency without increasing orientation errors.
Going, Going, Gone: Exploring Intention Communication for Multi-User Locomotion in Virtual Reality

Julian Rasch, Vladislav Dmitrievic Rusakov, Martin Schmitz, and Florian Müller

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

PDF ACM Video Talk

Exploring virtual worlds together with others adds a social component to the Virtual Reality (VR) experience that increases connectedness. In the physical world, joint locomotion comes naturally through implicit intention communication and subsequent adjustments of the movement patterns. In VR, however, discrete locomotion techniques such as point&teleport come without prior intention communication, hampering the collective experience. Related work proposes fixed groups, with a single person controlling the group movement, resulting in the loss of individual movement capabilities. To close the gap and mediate between these two extremes, we introduce three intention communication methods and explore them with two baseline methods. We contribute the results of a controlled experiment (n=20) investigating these methods from the perspective of a leader and a follower in a dyadic locomotion task. Our results suggest shared visualizations support the understanding of movement intentions, increasing the group feeling while maintaining individual freedom of movement.
Hands-On 3D Printed Electronics

Julian Rasch, Florian Müller, Thomas Kosch, Martin Schmitz, and Sebastian S. Feger

In Proceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction, Feb 2023

PDF ACM

The parallel improvements in multi-material 3D printers and the quality of conductive filament open new possibilities for the fabrication of tangible and functional objects. In this studio, we discuss best practices for 3D printed electronics, talk about encountered problems, and derive design recommendations. We will guide the participants through a fabrication process by practically designing and printing objects. Consequently, we contemplate individual functional fabricated components, including small printed circuits and multi-material prints. We aim to spark a discussion about individually experienced challenges participants encountered during their design and fabrication process. This discussion includes problemsolving strategies, whose insights benefit other participants. Finally, we show the potential of printed electronics and discuss encouraging new opportunities in this field.
Tailor Twist: Assessing Rotational Mid-Air Interactions for Augmented Reality

Dominik Schön, Thomas Kosch, Florian Müller, Martin Schmitz, Sebastian Günther, Lukas Bommhardt, and Max Mühlhäuser

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

PDF ACM Video Talk

Mid-air gestures, widely used in today’s Augmented Reality (AR) applications, are prone to the “gorilla arm” effect, leading to discomfort with prolonged interactions. While prior work has proposed metrics to quantify this effect and means to improve comfort and ergonomics, these works usually only consider simplistic, one-dimensional AR interactions, like reaching for a point or pushing a button. However, interacting with AR environments also involves far more complex tasks, such as rotational knobs, potentially impacting ergonomics. This paper advances the understanding of the ergonomics of rotational mid-air interactions in AR. For this, we contribute the results of a controlled experiment exposing the participants to a rotational task in the interaction space defined by their arms’ reach. Based on the results, we discuss how novel future mid-air gesture modalities benefit from our findings concerning ergonomic-aware rotational interaction.
DensingQueen: Exploration Methods for Spatial Dense Dynamic Data

Julius von Willich, Sebastian Günther, Andrii Matviienko, Martin Schmitz, Florian Müller, and Max Mühlhäuser

In Proceedings of the 2023 ACM Symposium on Spatial User Interaction, Oct 2023

PDF ACM Code

Research has proposed various interaction techniques to manage the occlusion of 3D data in Virtual Reality (VR), e.g., via gradual refinement. However, tracking dynamically moving data in a dense 3D environment poses the challenge of ever-changing occlusion, especially if motion carries relevant information, which is lost in still images. In this paper, we evaluated two interaction modalities for Spatial Dense Dynamic Data (SDDD), adapted from existing interaction methods for static and spatial data. We evaluated these modalities for exploring SDDD in VR, in an experiment with 18 participants. Furthermore, we investigated the influence of our interaction modalities on different levels of data density on the users’ performance in a no-knowledge task and a prior-knowledge task. Our results indicated significantly degraded performance for higher levels of density. Further, we found that our flashlight-inspired modality successfully improved tracking in SDDD, while a cutting plane-inspired approach was more suitable for highlighting static volumes of interest, particularly in such high-density environments.
Using Pseudo-Stiffness to Enrich the Haptic Experience in Virtual Reality

Yannick Weiss, Steeven Villa, Albrecht Schmidt, Sven Mayer, and Florian Müller

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

PDF ACM Talk

Providing users with a haptic sensation of the hardness and softness of objects in virtual reality is an open challenge. While physical props and haptic devices help, their haptic properties do not allow for dynamic adjustments. To overcome this limitation, we present a novel technique for changing the perceived stiffness of objects based on a visuo-haptic illusion. We achieved this by manipulating the hands’ Control-to-Display (C/D) ratio in virtual reality while pressing down on an object with fixed stiffness. In the first study (N=12), we determine the detection thresholds of the illusion. Our results show that we can exploit a C/D ratio from 0.7 to 3.5 without user detection. In the second study (N=12), we analyze the illusion’s impact on the perceived stiffness. Our results show that participants perceive the objects to be up to 28.1% softer and 8.9% stiffer, allowing for various haptic applications in virtual reality.

2022

Proper Posture: Designing Posture Feedback Across Musical Instruments

Bettina Eska, Jasmin Niess, and Florian Müller

May 2022

PDF arxiv

There is a recommended body posture and hand position for playing every musical instrument, allowing efficient and quick movements without blockage. Due to humans’ limited cognitive capabilities, they struggle to concentrate on several things simultaneously and thus sometimes lose the correct position while playing their instrument. Incorrect positions when playing an instrument can lead to injuries and movement disorders in the long run. Previous work in HCI mainly focused on developing systems to assist in learning an instrument. However, the design space for posture correction when playing a musical instrument has not yet been explored. In this position paper, we present our vision of providing subtle vibrotactile or thermal feedback to guide the focus of attention back to the correct posture when playing a musical instrument. We discuss our concept with a focus on motion recognition and feedback modalities. Finally, we outline the next steps for future research.
Smooth as Steel Wool: Effects of Visual Stimuli on the Haptic Perception of Roughness in Virtual Reality

Sebastian Günther, Julian Rasch, Dominik Schön, Florian Müller, Martin Schmitz, Jan Riemann, Andrii Matviienko, and Max Mühlhäuser

In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Apr 2022

PDF ACM Video Talk

Haptic Feedback is essential for lifelike Virtual Reality (VR) experiences. To provide a wide range of matching sensations of being touched or stroked, current approaches typically need large numbers of different physical textures. However, even advanced devices can only accommodate a limited number of textures to remain wearable. Therefore, a better understanding is necessary of how expectations elicited by different visualizations affect haptic perception, to achieve a balance between physical constraints and great variety of matching physical textures. In this work, we conducted an experiment (N=31) assessing how the perception of roughness is affected within VR. We designed a prototype for arm stroking and compared the effects of different visualizations on the perception of physical textures with distinct roughnesses. Additionally, we used the visualizations’ real-world materials, no-haptics and vibrotactile feedback as baselines. As one result, we found that two levels of roughness can be sufficient to convey a realistic illusion.
NotiBike: Assessing Target Selection Techniques for Cyclist Notifications in Augmented Reality

Thomas Kosch, Andrii Matviienko, Florian Müller, Jessica Bersch, Christopher Katins, Dominik Schön, and Max Mühlhäuser

Proceedings of the ACM on Human-Computer Interaction, Sep 2022

PDF ACM Video

Cyclists’ attention is often compromised when interacting with notifications in traffic, hence increasing the likelihood of road accidents. To address this issue, we evaluate three notification interaction modalities and investigate their impact on the interaction performance while cycling: gaze-based Dwell Time, Gestures, and Manual And Gaze Input Cascaded (MAGIC) Pointing. In a user study (N=18), participants confirmed notifications in Augmented Reality (AR) using the three interaction modalities in a simulated biking scenario. We assessed the efficiency regarding reaction times, error rates, and perceived task load. Our results show significantly faster response times for MAGIC Pointing compared to Dwell Time and Gestures, while Dwell Time led to a significantly lower error rate compared to Gestures. Participants favored the MAGIC Pointing approach, supporting cyclists in AR selection tasks. Our research sets the boundaries for more comfortable and easier interaction with notifications and discusses implications for target selections in AR while cycling.
SkyPort: Investigating 3D Teleportation Methods in Virtual Environments

Andrii Matviienko, Florian Müller, Martin Schmitz, Marco Fendrich, and Max Mühlhäuser

In CHI Conference on Human Factors in Computing Systems, Apr 2022

PDF ACM Talk

Teleportation has become the de facto standard of locomotion in Virtual Reality (VR) environments. However, teleportation with parabolic and linear target aiming methods is restricted to horizontal 2D planes and it is unknown how they transfer to the 3D space. In this paper, we propose six 3D teleportation methods in virtual environments based on the combination of two existing aiming methods (linear and parabolic) and three types of transitioning to a target (instant, interpolated and continuous). To investigate the performance of the proposed teleportation methods, we conducted a controlled lab experiment (N = 24) with a mid-air coin collection task to assess accuracy, efficiency and VR sickness. We discovered that the linear aiming method leads to faster and more accurate target selection. Moreover, a combination of linear aiming and instant transitioning leads to the highest efficiency and accuracy without increasing VR sickness.
Baby, You Can Ride My Bike: Exploring Maneuver Indications of Self-Driving Bicycles Using a Tandem Simulator

Andrii Matviienko, Damir Mehmedovic, Florian Müller, and Max Mühlhäuser

Proceedings of the ACM on Human-Computer Interaction, Sep 2022

PDF ACM Video

We envision a future where self-driving bicycles can take us to our destinations. This allows cyclists to use their time on the bike efficiently for work or relaxation without having to focus their attention on traffic. In the related field of self-driving cars, research has shown that communicating the planned route to passengers plays an important role in building trust in automation and situational awareness. For self-driving bicycles, this information transfer will be even more important, as riders will need to actively compensate for the movement of a self-driving bicycle to maintain balance. In this paper, we investigate maneuver indications for self-driving bicycles: (1) ambient light in a helmet, (2) head-up display indications, (3) speech feedback, (4) vibration on the handlebar, and (5) no assistance. To evaluate these indications, we conducted an outdoor experiment (N = 25) in a proposed tandem simulator consisting of a tandem bicycle with a steering and braking control on the back seat and a rider in full control of it. Our results indicate that riders respond faster to visual cues and focus comparably on the reading task while riding with and without maneuver indications. Additionally, we found that the tandem simulator is realistic, safe, and creates an awareness of a human cyclist controlling the tandem.
BikeAR: Understanding Cyclists’ Crossing Decision-Making at Uncontrolled Intersections Using Augmented Reality

Andrii Matviienko, Florian Müller, Dominik Schön, Paul Seesemann, Sebastian Günther, and Max Mühlhäuser

In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Apr 2022

PDF ACM Video Talk

Cycling has become increasingly popular as a means of transportation. However, cyclists remain a highly vulnerable group of road users. According to accident reports, one of the most dangerous situations for cyclists are uncontrolled intersections, where cars approach from both directions. To address this issue and assist cyclists in crossing decision-making at uncontrolled intersections, we designed two visualizations that: (1) highlight occluded cars through an X-ray vision and (2) depict the remaining time the intersection is safe to cross via a Countdown. To investigate the efficiency of these visualizations, we proposed an Augmented Reality simulation as a novel evaluation method, in which the above visualizations are represented as AR, and conducted a controlled experiment with 24 participants indoors. We found that the X-ray ensures a fast selection of shorter gaps between cars, while the Countdown facilitates a feeling of safety and provides a better intersection overview.
E-ScootAR: Exploring Unimodal Warnings for E-Scooter Riders in Augmented Reality

Andrii Matviienko, Florian Müller, Dominik Schön, Régis Fayard, Salar Abaspur, Yi Li, and Max Mühlhäuser

In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, Apr 2022

PDF ACM Talk

Micro-mobility is becoming a more popular means of transportation. However, this increased popularity brings its challenges. In particular, the accident rates for E-Scooter riders increase, which endangers the riders and other road users. In this paper, we explore the idea of augmenting E-Scooters with unimodal warnings to prevent collisions with other road users, which include Augmented Reality (AR) notifications, vibrotactile feedback on the handlebar, and auditory signals in the AR glasses. We conducted an outdoor experiment (N = 13) using an Augmented Reality simulation and compared these types of warnings in terms of reaction time, accident rate, and feeling of safety. Our results indicate that AR and auditory warnings lead to shorter reaction times, have a better perception, and create a better feeling of safety than vibrotactile warnings. Moreover, auditory signals have a higher acceptance by the riders compared to the other two types of warnings.
Reducing Virtual Reality Sickness for Cyclists in VR Bicycle Simulators

Andrii Matviienko, Florian Müller, Marcel Zickler, Lisa Alina Gasche, Julia Abels, Till Steinert, and Max Mühlhäuser

In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Apr 2022

PDF ACM Talk

Virtual Reality (VR) bicycle simulations aim to recreate the feeling of riding a bicycle and are commonly used in many application areas. However, current solutions still create mismatches between the visuals and physical movement, which causes VR sickness and diminishes the cycling experience. To reduce VR sickness in bicycle simulators, we conducted two controlled lab experiments addressing two main causes of VR sickness: (1) steering methods and (2) cycling trajectory. In the first experiment (N = 18) we compared handlebar, HMD, and upper-body steering methods. In the second experiment (N = 24) we explored three types of movement in VR (1D, 2D, and 3D trajectories) and three countermeasures (airflow, vibration, and dynamic Field-of-View) to reduce VR sickness. We found that handlebar steering leads to the lowest VR sickness without decreasing cycling performance and airflow suggests to be the most promising method to reduce VR sickness for all three types of trajectories.
MuC ’22: Proceedings of Mensch Und Computer 2022

Max Mühlhauser, Christian Reuter, Bastian Pfleging, Thomas Kosch, Andrii Matviienko, Kathrin Gerling, Sven Mayer, Wilko Heuten, Tanja Döring, Florian Müller, and Martin Schmitz

Sep 2022

ACM

Die Mensch und Computer 2022 fand vom 4.-7. September 2022 an der Technischen Universität Darmstadt statt. Die 2001 ins Leben gerufene ,,Mensch und Computer“ (MuC) ist die größte Tagungsreihe der Mensch-Computer-Interaktion in Europa. Wir sind dankbar, dass wir im Jahr des 50jährigen Bestehens des Fachbereichs Informatik der TU Darmstadt fast 600 Teilnehmende aus Wissenschaft und Industrie live, in Farbe, physisch, also real in Darmstadt begrüßen konnten – passend zu unserem Konferenzmotto: ,,Facing Realities“. Träger der Konferenz waren der Fachbereich Mensch-Computer-Interaktion der Gesellschaft für Informatik (GI e.V.) und die German UPA – der Berufsverband der Deutschen Usability und User Experience Professionals. Die Technische Universität Darmstadt und insbesondere der Fachbereich Informatik fungierte als Gastgeber.
Squeezy-Feely: Investigating Lateral Thumb-Index Pinching as an Input Modality

Martin Schmitz, Sebastian Günther, Dominik Schön, and Florian Müller

In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Apr 2022

PDF ACM Video Talk

From zooming on smartphones and mid-air gestures to deformable user interfaces, thumb-index pinching grips are used in many interaction techniques. However, there is still a lack of systematic understanding of how the accuracy and efficiency of such grips are affected by various factors such as counterforce, grip span, and grip direction. Therefore, in this paper, we contribute an evaluation (N = 18) of thumb-index pinching performance in a visual targeting task using scales up to 75 items. As part of our findings, we conclude that the pinching interaction between the thumb and index finger is a promising modality also for one-dimensional input on higher scales. Furthermore, we discuss and outline implications for future user interfaces that benefit from pinching as an additional and complementary interaction modality.
Rethinking Smart Objects: The International Workshop on Interacting with Smart Objects in Interactive Spaces

Martin Schmitz, Sebastian Günther, Karola Marky, Florian Müller, Andrii Matviienko, Alexandra Voit, Roberts Marky, Max Mühlhäuser, and Thomas Kosch

In Companion Proceedings of the 2022 Conference on Interactive Surfaces and Spaces, Nov 2022

PDF ACM Website

The increasing proliferation of smart objects in everyday life has changed how we interact with computers. Instead of concentrating computational capabilities and interaction into one device, everyday objects have naturally integrated parts of interactive features. Although this has led to many practical applications, the possibilities for explicit or implicit interaction with such objects are still limited in interaction spaces. We still often rely on smartphones as interactive hubs for controlling smart objects, hence not fulfilling the vision of truly smart objects. The workshop Rethinking Smart Objects invites practitioners and researchers from both academia and industry to discuss novel interaction paradigms and the integration and societal implications of using smart objects in interactive space. This workshop will include an action plan with leading questions, aiming to move the research field forward.
TrackItPipe: A Fabrication Pipeline To Incorporate Location and Rotation Tracking Into 3D Printed Objects

Dominik Schön, Thomas Kosch, Martin Schmitz, Florian Müller, Sebastian Günther, Johannes Kreutz, and Max Mühlhäuser

In Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, Oct 2022

PDF ACM Code

The increasing convergence of the digital and physical world creates a growing urgency to integrate 3D printed physical tangibles with virtual environments. A precise position and rotation tracking are essential to integrate such physical objects with a virtual environment. However, available 3D models commonly do not provide tracking support on their composition, which requires modifications by CAD experts. This poses a challenge for users with no prior CAD experience. This work presents TrackItPipe, a fabrication pipeline supporting users by semi-automatically adding tracking capabilities for 3D printable tangibles tailored to environmental requirements. TrackItPipe integrates modifications to the 3D model, produces the respective tangibles for 3D printing, and provides integration scripts for Mixed Reality. Using TrackItPipe, users can rapidly equip objects with tracking capabilities.

2021

CameraReady: Assessing the Influence of Display Types and Visualizations on Posture Guidance

Hesham Elsayed, Philipp Hoffmann, Sebastian Günther, Martin Schmitz, Martin Weigel, Max Mühlhäuser, and Florian Müller

In Designing Interactive Systems Conference 2021, Jun 2021

PDF ACM Video Talk

Computer-supported posture guidance is used in sports, dance training, expression of art with movements, and learning gestures for interaction. At present, the influence of display types and visualizations have not been investigated in the literature. These factors are important as they directly impact perception and cognitive load, and hence influence the performance of participants. In this paper, we conducted a controlled experiment with 20 participants to compare the use of five display types with different screen sizes: smartphones, tablets, desktop monitors, TVs, and large displays. On each device, we compared three common visualizations for posture guidance: skeletons, silhouettes, and 3d body models. To conduct our assessment, we developed a mobile and cross-platform system that only requires a single camera. Our results show that compared to a smartphone display, larger displays show a lower error (12%). Regarding the choice of visualization, participants rated 3D body models as significantly more usable in comparison to a skeleton visualization.
ActuBoard: An Open Rapid Prototyping Platform to Integrate Hardware Actuators in Remote Applications

Sebastian Günther, Florian Müller, Felix Hübner, Max Mühlhäuser, and Andrii Matviienko

In Companion of the 2021 ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Jun 2021

PDF ACM Code

Prototyping is an essential step in developing tangible experiences and novel devices, ranging from haptic feedback to wearables. However, prototyping of actuated devices nowadays often requires repetitive and time-consuming steps, such as wiring, soldering, and programming basic communication, before HCI researchers and designers can focus on their primary interest: designing interaction. In this paper, we present ActuBoard, a prototyping platform to support 1) quick assembly, 2) less preparation work, and 3) the inclusion of non-tech-savvy users. With ActuBoard, users are not required to create complex circuitry, write a single line of firmware, or implementing communication protocols. Acknowledging existing systems, our platform combines the flexibility of low-level microcontrollers and ease-of-use of abstracted tinker platforms to control actuators from separate applications. As further contribution, we highlight the technical specifications and published the ActuBoard platform as Open Source.
Let’s Frets! Assisting Guitar Students During Practice via Capacitive Sensing

Karola Marky, Andreas Weiß, Andrii Matviienko, Florian Brandherm, Sebastian Wolf, Martin Schmitz, Florian Krell, Florian Müller, Max Mühlhäuser, and Thomas Kosch

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, May 2021

PDF ACM Video Talk

Learning a musical instrument requires regular exercise. However, students are often on their own during their practice sessions due to the limited time with their teachers, which increases the likelihood of mislearning playing techniques. To address this issue, we present Let’s Frets - a modular guitar learning system that provides visual indicators and capturing of finger positions on a 3D-printed capacitive guitar fretboard. We based the design of Let’s Frets on requirements collected through in-depth interviews with professional guitarists and teachers. In a user study (N=24), we evaluated the feedback modules of Let’s Frets against fretboard charts. Our results show that visual indicators require the least time to realize new finger positions while a combination of visual indicators and position capturing yielded the highest playing accuracy. We conclude how Let’s Frets enables independent practice sessions that can be translated to other musical instruments.
Let’s Frets! Mastering Guitar Playing with Capacitive Sensing and Visual Guidance

Karola Marky, Andreas Weiß, Florian Müller, Martin Schmitz, Max Mühlhäuser, and Thomas Kosch

In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, May 2021

PDF ACM Video

Mastering the guitar requires regular exercise to develop new skills and maintain existing abilities. We present Let’s Frets - a modular guitar support system that provides visual guidance through LEDs that are integrated into a capacitive fretboard to support the practice of chords, scales, melodies, and exercises. Additional feedback is provided through a 3D-printed fretboard that senses the finger positions through capacitive sensing. We envision Let’s Frets as an integrated guitar support system that raises the awareness of guitarists about their playing styles, their training progress, the composition of new pieces, and facilitating remote collaborations between teachers as well as guitar students. This interactivity demonstrates Let’s Frets with an augmented fretboard and supporting software that runs on a mobile device.
VRtangibles: Assisting Children in Creating Virtual Scenes Using Tangible Objects and Touch Input

Andrii Matviienko, Marcel Langer, Florian Müller, Martin Schmitz, and Max Mühlhäuser

In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, May 2021

PDF ACM Talk

Children are increasingly exposed to virtual reality (VR) technology as end-users. However, they miss an opportunity to become active creators due to the barrier of insufficient technical background. Creating scenes in VR requires considerable programming knowledge and excludes non-tech-savvy users, e.g., school children. In this paper, we showcase a system called VRtangibles, which combines tangible objects and touch input to create virtual scenes without programming. With VRtangibles, we aim to engage children in the active creation of virtual scenes via playful hands-on activities. From the lab study with six school children, we discovered that the majority of children were successful in creating virtual scenes using VRtangibles and found it engaging and fun to use.
Itsy-Bits: Fabrication and Recognition of 3D-Printed Tangibles with Small Footprints on Capacitive Touchscreens

Martin Schmitz, Florian Müller, Max Mühlhäuser, Jan Riemann, and Huy Viet Viet Le

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, May 2021

PDF ACM Video Talk

Tangibles on capacitive touchscreens are a promising approach to overcome the limited expressiveness of touch input. While research has suggested many approaches to detect tangibles, the corresponding tangibles are either costly or have a considerable minimal size. This makes them bulky and unattractive for many applications. At the same time, they obscure valuable display space for interaction. To address these shortcomings, we contribute Itsy-Bits: a fabrication pipeline for 3D printing and recognition of tangibles on capacitive touchscreens with a footprint as small as a fingertip. Each Itsy-Bit consists of an enclosing 3D object and a unique conductive 2D shape on its bottom. Using only raw data of commodity capacitive touchscreens, Itsy-Bits reliably identifies and locates a variety of shapes in different sizes and estimates their orientation. Through example applications and a technical evaluation, we demonstrate the feasibility and applicability of Itsy-Bits for tangibles with small footprints.
Oh, Snap! A Fabrication Pipeline to Magnetically Connect Conventional and 3D-Printed Electronics

Martin Schmitz, Jan Riemann, Florian Müller, Steffen Kreis, and Max Mühlhäuser

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, May 2021

PDF ACM Video Talk Code

3D printing has revolutionized rapid prototyping by speeding up the creation of custom-shaped objects. With the rise of multi-material 3D printers, these custom-shaped objects can now be made interactive in a single pass through passive conductive structures. However, connecting conventional electronics to these conductive structures often still requires time-consuming manual assembly involving many wires, soldering or gluing. To alleviate these shortcomings, we propose : a fabrication pipeline and interfacing concept to magnetically connect a 3D-printed object equipped with passive sensing structures to conventional sensing electronics. To this end, utilizes ferromagnetic and conductive 3D-printed structures, printable in a single pass on standard printers. We further present a proof-of-concept capacitive sensing board that enables easy and robust magnetic assembly to quickly create interactive 3D-printed objects. We evaluate by assessing the robustness and quality of the connection and demonstrate its broad applicability by a series of example applications.

2020

VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments

Hesham Elsayed, Mayra Donaji Barrera Machuca, Christian Schaarschmidt, Karola Marky, Florian Müller, Jan Riemann, Andrii Matviienko, Martin Schmitz, Martin Weigel, and Max Mühlhäuser

In 26th ACM Symposium on Virtual Reality Software and Technology, Nov 2020

PDF ACM

Accurate sketching in virtual 3D environments is challenging due to aspects like limited depth perception or the absence of physical support. To address this issue, we propose VRSketchPen - a pen that uses two haptic modalities to support virtual sketching without constraining user actions: (1) pneumatic force feedback to simulate the contact pressure of the pen against virtual surfaces and (2) vibrotactile feedback to mimic textures while moving the pen over virtual surfaces. To evaluate VRSketchPen, we conducted a lab experiment with 20 participants to compare (1) pneumatic, (2) vibrotactile and (3) a combination of both with (4) snapping and no assistance for flat and curved surfaces in a 3D virtual environment. Our findings show that usage of pneumatic, vibrotactile and their combination significantly improves 2D shape accuracy and leads to diminished depth errors for flat and curved surfaces. Qualitative results indicate that users find the addition of unconstraining haptic feedback to significantly improve convenience, confidence and user experience.
VibroMap: Understanding the Spacing of Vibrotactile Actuators across the Body

Hesham Elsayed, Martin Weigel, Florian Müller, Martin Schmitz, Karola Marky, Sebastian Günther, Jan Riemann, and Max Mühlhäuser

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Dec 2020

PDF ACM Talk

In spite of the great potential of on-body vibrotactile displays for a variety of applications, research lacks an understanding of the spacing between vibrotactile actuators. Through two experiments, we systematically investigate vibrotactile perception on the wrist, forearm, upper arm, back, torso, thigh, and leg, each in transverse and longitudinal body orientation. In the first experiment, we address the maximum distance between vibration motors that still preserves the ability to generate phantom sensations. In the second experiment, we investigate the perceptual accuracy of localizing vibrations in order to establish the minimum distance between vibration motors. Based on the results, we derive VibroMap, a spatial map of the functional range of inter-motor distances across the body. VibroMap supports hardware and interaction designers with design guidelines for constructing body-worn vibrotactile displays.
Therminator : Understanding the Interdependency of Visual and On-Body Thermal Feedback in Virtual Reality

Sebastian Günther, Omar Elmoghazy, Florian Müller, Max Mühlhäuser, Dominik Schön, and Martin Schmitz

In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems - CHI ’20, Dec 2020

PDF ACM Video Talk

Recent advances have made Virtual Reality (VR) more realistic than ever before. This improved realism is attributed to today’s ability to increasingly appeal to human sensations, such as visual, auditory or tactile. While research also examines temperature sensation as an important aspect, the interdependency of visual and thermal perception in VR is still underexplored. In this paper, we propose Therminator, a thermal display concept that provides warm and cold on-body feedback in VR through heat conduction of flowing liquids with different temperatures. Further, we systematically evaluate the interdependency of different visual and thermal stimuli on the temperature perception of arm and abdomen with 25 participants. As part of the results, we found varying temperature perception depending on the stimuli, as well as increasing involvement of users during conditions with matching stimuli.
PneumoVolley: Pressure-based Haptic Feedback on the Head through Pneumatic Actuation

Sebastian Günther, Dominik Schön, Florian Müller, Max Mühlhäuser, and Martin Schmitz

In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems - CHI ’20, Dec 2020

PDF ACM Video Talk

Haptic Feedback brings immersion and presence in Virtual Reality (VR) to the next level. While research proposes the usage of various tactile sensations, such as vibration or ultrasound approaches, the potential applicability of pressure feedback on the head is still under-explored. In this paper, we contribute concepts and design considerations for pressure-based feedback on the head through pneumatic actuation. As a proof-of-concept implementing our pressure-based haptics, we further present PneumoVolley: a VR experience similar to the classic Volleyball game but played with the head. In an exploratory user study with 9 participants, we evaluated our concepts and identified a significantly increased involvement compared to a no-haptics baseline along with high realism and enjoyment ratings using pressure-based feedback on the head in VR. LBW033, Page 1 CHI 2020 Late-Breaking Work
Exploring User Expectations of Proactive AI Systems

Christian Meurisch, Cristina A. Mihale-Wilson, Adrian Hawlitschek, Florian Giger, Florian Müller, Oliver Hinz, and Max Mühlhäuser

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Dec 2020

PDF ACM

Recent advances in artificial intelligence (AI) enabled digital assistants to evolve towards proactive user support. However, expectations as to when and to what extent assistants should take the initiative are still unclear; discrepancies to the actual system behavior might negatively affect user acceptance. In this paper, we present an in-the-wild study for exploring user expectations of such user-supporting AI systems in terms of different proactivity levels and use cases. We collected 3,168 in-situ responses from 272 participants through a mixed method of automated user tracking and context-triggered surveying. Using a data-driven approach, we gain insights into initial expectations and how they depend on different human factors and contexts. Our insights can help to design AI systems with varying degree of proactivity and preset to meet individual expectations.
Around-Body Interaction: Leveraging Limb-movements for Interacting in a Digitally Augmented Physical World

Florian Müller

Dec 2020

PDF TUprints

Recent technological advances have made head-mounted displays (HMDs) smaller and untethered, fostering the vision of ubiquitous interaction with information in a digitally augmented physical world. For interacting with such devices, three main types of input - besides not very intuitive finger gestures - have emerged so far: 1) Touch input on the frame of the devices or 2) on accessories (controller) as well as 3) voice input. While these techniques have both advantages and disadvantages depending on the current situation of the user, they largely ignore the skills and dexterity that we show when interacting with the real world: Throughout our lives, we have trained extensively to use our limbs to interact with and manipulate the physical world around us. This thesis explores how the skills and dexterity of our upper and lower limbs, acquired and trained in interacting with the real world, can be transferred to the interaction with HMDs. Thereby, this thesis develops the vision of around-body interaction, in which we use the space around our body, defined by the reach of our limbs, for fast, accurate, and enjoyable interaction with information.
Around-Body Interaction: Interacting While on the Go

Florian Müller, Sebastian Günther, and Max Mühlhauser

IEEE Pervasive Computing, Apr 2020

IEEE

We provide a concise overview on how the skills and dexterity of our upper and lower limbs, acquired and trained in interacting with the physical world, can be transferred to the interaction with HMD. We present the vision of around-body interaction, in which we use the space around our body, defined by the reach of our limbs, for fast, accurate, and enjoyable interactions. In the remainder of this article, we first introduce a design space for such around-body interactions and, second, present two examples of such interaction techniques, which use the degrees of freedom of the lower limbs for on-the go interactions. Finally, we conclude with future research challenges.
Around-Body Interaction: Über Die Nutzung Der Bewegungen von Gliedmaßen Zur Interaktion in Einer Digital Erweiterten Physischen Welt

Florian Müller

In Ausgezeichnete Informatikdissertationen 2019, May 2020

PDF

Durch den technischen Fortschritt sind Head-Mounted Displays (HMDs) kleiner und kabellos geworden. Sie leisten so einen Beitrag zur Vision von allgegenwärtiger Interaktion mit Informationen in einer digital erweiterten physischen Welt. Zur Interaktion mit solchen Geräten werden heute eingabeseitig - neben wenig intuitiven Fingergesten in der Luft – vor allem dreierlei Techniken verwendet: 1) Toucheingabe auf dem Gehäuse der Geräte oder 2) auf Zubehör (Controller) sowie 3) Spracheingabe. Während diese Techniken, abhängig von der aktuellen Situation des Benutzers, sowohl Vor- als auch Nachteile haben, so ignorieren sie weitgehend die Fähigkeiten und Geschicklichkeit, die wir im Umgang mit der realen Welt zeigen: Während unseres ganzen Lebens haben wir ausgiebig trainiert unsere Gliedmaßen zu benutzen, um mit der physischen Welt um uns herum zu interagieren und sie zu manipulieren. Diese Arbeit entwickelt eine Vision für eine körperlichere Interaktion mit solchen Geräten, welche die Fähigkeiten und Geschicklichkeit, die wir im Umgang mit der physischen Welt zeigen, auf die Interaktion mit HMDs überträgt.
Walk The Line: Leveraging Lateral Shifts of the Walking Path as an Input Modality for Head-Mounted Displays

Florian Müller, Martin Schmitz, Daniel Schmitt, Sebastian Günther, Markus Funk, and Max Mühlhäuser

In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Apr 2020

PDF ACM Video Talk

Recent technological advances have made head-mounted displays (HMDs) smaller and untethered, fostering the vision of ubiquitous interaction in a digitally augmented physical world. Consequently, a major part of the interaction with such devices will happen on the go, calling for interaction techniques that allow users to interact while walking. In this paper, we explore lateral shifts of the walking path as a hands-free input modality. The available input options are visualized as lanes on the ground parallel to the user’s walking path. Users can select options by shifting the walking path sideways to the respective lane. We contribute the results of a controlled experiment with 18 participants, confirming the viability of our approach for fast, accurate, and joyful interactions. Further, based on the findings of the controlled experiment, we present three example applications.
Podoportation: Foot-Based Locomotion in Virtual Reality

Julius Von Willich, Martin Schmitz, Florian Müller, Daniel Schmitt, and Max Mühlhäuser

In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems - CHI ’20, Apr 2020

PDF ACM Video Talk

Virtual Reality (VR) allows for infinitely large environments. However, the physical traversable space is always limited by real-world boundaries. This discrepancy between physical and virtual dimensions renders traditional locomotion methods used in real world unfeasible. To alleviate these limitations, research proposed various artificial locomotion concepts such as teleportation, treadmills, and redirected walking. However, these concepts occupy the user’s hands, require complex hardware or large physical spaces. In this paper, we contribute nine VR locomotion concepts for foot-based locomotion, relying on the 3D position of the user’s feet and the pressure applied to the sole as input modalities. We evaluate our concepts and compare them to state-of-the-art point & teleport technique in a controlled experiment with 20 participants. The results confirm the viability of our approaches for foot-based and engaging locomotion. Further, based on the findings, we contribute a wireless hardware prototype implementation.

2019

Trends on Engineering Interactive Systems: An Overview of Works Presented in Workshops at EICS 2019

Damiano Distante, Alexandra Voit, Marco Winckler, Regina Bernhaupt, Judy Bowen, José Creissac Campos, Florian Müller, Philippe Palanque, Jan Van den Bergh, and Benjamin Weyers

In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS ’19, Apr 2019

PDF ACM

Workshops are a great opportunity for identifying innovative topics of research that might require discussion and maturation. This paper summarizes the outcomes of the workshops track of the 11th Engineering Interactive Computing Systems conference (EICS 2019), held in Valencia (Spain) on 18-21 June 2019. The track featured three workshops, one half-day, one full-day and one two-days workshop, each focused on specific topics of the ongoing research in engineering usable and effective interactive computing systems. In particular, the list of discussed topics include novel forms of interaction and emerging themes in HCI related to new application domains, more efficient and enjoyable interaction possibilities associated to smart objects and smart environments, challenges faced in designing, developing and using interactive systems involving multiple stakeholders.
Assessing the Accuracy of Point & Teleport Locomotion with Orientation Indication for Virtual Reality Using Curved Trajectories

Markus Funk, Florian Müller, Marco Fendrich, Megan Shene, Moritz Kolvenbach, Niclas Dobbertin, Sebastian Günther, and Max Mühlhäuser

In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, May 2019

PDF ACM Video

Room-scale Virtual Reality (VR) systems have arrived in users’ homes where tracked environments are set up in limited physical spaces. As most Virtual Environments (VEs) are larger than the tracked physical space, locomotion techniques are used to navigate in VEs. Currently, in recent VR games, point & teleport is the most popular locomotion technique. However, it only allows users to select the position of the teleportation and not the orientation that the user is facing after the teleport. This results in users having to manually correct their orientation after teleporting and possibly getting entangled by the cable of the headset. In this paper, we introduce and evaluate three different point & teleport techniques that enable users to specify the target orientation while teleporting. The results show that, although the three teleportation techniques with orientation indication increase the average teleportation time, they lead to a decreased need for correcting the orientation after teleportation.
Slappyfications: Towards Ubiquitous Physical and Embodied Notifications

Sebastian Günther, Florian Müller, Markus Funk, and Max Mühlhäuser

In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, May 2019

PDF ACM Video

With emerging trends of notifying persons through ubiquitous technologies [2], such as ambient light, vibrotactile, or auditory cues, none of these technologies are truly ubiquitous and have proven to be easily missed or ignored. In this work, we propose Slappyfications, a novel way of sending unmissable embodied and ubiquitous notifications through a palm-based interface [1]. Our prototype enables the users to send three types of Slappyfications: poke, slap, and the STEAM-HAMMER. Through a Wizard-of-Oz study, we show the applicability of our system in real-world scenarios. The results reveal a promising trend, as none of the participants missed a single Slappyfication.
PneumAct: Pneumatic Kinesthetic Actuation of Body Joints in Virtual Reality Environments

Sebastian Günther, Mohit Makhija, Florian Müller, Dominik Schön, Max Mühlhäuser, and Markus Funk

In Proceedings of the 2019 on Designing Interactive Systems Conference, Jun 2019

PDF ACM Video

Virtual Reality Environments (VRE) create an immersive user experience through visual, aural, and haptic sensations. However, the latter is often limited to vibrotactile sensations that are not able to actively provide kinesthetic motion actuation. Further, such sensations do not cover natural representations of physical forces, for example, when lifting a weight. We present PneumAct, a jacket to enable pneumatically actuated kinesthetic movements of arm joints in VRE. It integrates two types of actuators inflated through compressed air: a Contraction Actuator and an Extension Actuator. We evaluate our PneumAct jacket through two user studies with a total of 32 participants: First, we perform a technical evaluation measuring the contraction and extension angles of different inflation patterns and inflation durations. Second, we evaluate PneumAct in three VRE scenarios comparing our system to traditional controller-based vibrotactile and a baseline without haptic feedback.
Mind the Tap: Assessing Foot-Taps for Interacting with Head-Mounted Displays

Florian Müller, Joshua McManus, Sebastian Günther, Martin Schmitz, Max Mühlhäuser, and Markus Funk

In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, Jun 2019

PDF ACM Video

From voice commands and air taps to touch gestures on frames: Various techniques for interacting with head-mounted displays (HMDs) have been proposed. While these techniques have both benefits and drawbacks dependent on the current situation of the user, research on interacting with HMDs has not concluded yet. In this paper, we add to the body of research on interacting with HMDs by exploring foot-tapping as an input modality. Through two controlled experiments with a total of 36 participants, we first explore direct interaction with interfaces that are displayed on the floor and require the user to look down to interact. Secondly, we investigate indirect interaction with interfaces that, although operated by the user’s feet, are always visible as they are floating in front of the user. Based on the results of the two experiments, we provide design recommendations for direct and indirect foot-based user interfaces.
Proceedings of the 7th Workshop on Interacting with Smart Objects

Jun 2019

PDF

These are the proceedings of the 7th Workshop on Interacting with Smart Objects (SmartObjects ’19) in conjunction with EICS’19 held on June 18, 2019, in Valencia, Spain. This volume contains the ten accepted papers. Each submission was reviewed by three program committee members. Objects that we use in our everyday life are ever-expanding their interaction capabilities and provide functionalities that go far beyond their original functionality. They feature computing capabilities and are, thus, able to capture, process and store information and interact with their environments, turning them into smart objects. Their wide range was covered by the submissions to this workshop. Smart objects know something about their users and, thus, allow for natural interaction. Natural interaction, in contrast, does not imply smartness. Smartness requires interaction with users and provides help. There are already commercialized products available that expose their properties and interaction capabilities. To enrich their potential and to lower affordances, they need to communicate to each other. Making sense out of the available data in this field is still an open research question. The overall goal should be to build an interactive ecosystem that (i) seamlessly discovers, connects and talks to its environment, (ii) is ubiquitous and (iii) allows the user to be in control. The workshop examined these issues with regards to the following aspects: ∙ Smart Devices ∙ Smart Spaces Putting together SmartObjects ‘19 was a team effort. We would like to send out our thanks to everybody who has helped us to organize this event: ∙ The authors, who have written and submitted their papers to the workshop. ∙ The program committee and the external reviewers, for their time and effort to write substantial and constructive review reports. We hope that you will find this program interesting and thought-provoking and that the workshop will provide you with a valuable opportunity to share ideas with other researchers and practitioners from institutions around the world.
./Trilaterate: A Fabrication Pipeline to Design and 3D Print Hover-, Touch-, and Force-Sensitive Objects

Martin Schmitz, Martin Stitz, Florian Müller, Markus Funk, and Max Mühlhäuser

In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, Jun 2019

PDF ACM Video

Hover, touch, and force are promising input modalities that get increasingly integrated into screens and everyday objects. However, these interactions are often limited to fat surfaces and the integration of suitable sensors is time-consuming and costly. To alleviate these limitations, we contribute Tri-laterate: A fabrication pipeline to 3D print custom objects that detect the 3D position of a fnger hovering, touching, or forcing them by combining multiple capacitance measurements via capacitive trilateration. Trilaterate places and routes actively-shielded sensors inside the object and operates on consumer-level 3D printers. We present technical evaluations and example applications that validate and demonstrate the wide applicability of Trilaterate. CCS CONCEPTS ∙ Human-centered computing → Interaction devices; ∙ Hardware → Tactile and hand-based interfaces;
VRChairRacer: Using an Office Chair Backrest as a Locomotion Technique for VR Racing Games

Julius von Willich, Dominik Schön, Sebastian Günther, Florian Müller, Max Mühlhäuser, and Markus Funk

In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, Jun 2019

PDF ACM Video

Locomotion in Virtual Reality (VR) is an important topic as there is a mismatch between the size of a Virtual Environment and the physically available tracking space. Although many locomotion techniques have been proposed, research on VR locomotion has not concluded yet. In this demonstration, we contribute to the area of VR locomotion by introducing VRChairRacer. VRChairRacer introduces a novel mapping the velocity of a racing cart on the backrest of an office chair. Further, it maps a users’ rotation onto the steering of a virtual racing cart. VRChairRacer demonstrates this locomotion technique to the community through an immersive multiplayer racing demo.
You Invaded My Tracking Space! Using Augmented Virtuality for Spotting Passersby in Room-Scale Virtual Reality

Julius von Willich, Markus Funk, Florian Müller, Karola Marky, Jan Riemann, and Max Mühlhäuser

In Proceedings of the 2019 on Designing Interactive Systems Conference - DIS ’19, Jun 2019

PDF ACM Video

With the proliferation of room-scale Virtual Reality (VR), more and more users install a VR system in their homes. When users are in VR, they are usually completely immersed in their application. However, sometimes passersby invade these tracking spaces and walk up to users that are currently immersed in VR to try and interact with them. As this either scares the user in VR or breaks the user’s immersion, research has yet to find a way to seamlessly represent physical passersby in virtual worlds. In this paper, we propose and evaluate three different ways to represent physical passersby in a Virtual Environment using Augmented Virtuality. The representations encompass showing a Pointcloud, showing a 3D-Model, and showing an Image Overlay of the passerby. Our results show that while an Image Overlay and a 3D-Model are the fastest representations to spot passersby, the 3D-Model and the Pointcloud representations were the most accurate.

2018

CheckMate: Exploring a Tangible Augmented Reality Interface for Remote Interaction

Sebastian Günther, Florian Müller, Martin Schmitz, Jan Riemann, Niloofar Dezfuli, Markus Funk, Dominik Schön, and Max Mühlhäuser

In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Apr 2018

PDF ACM Video

The digitalized world comes with increasing Internet capabilities, allowing to connect persons over distance easier than ever before. Video conferencing and similar online applications create great benefits bringing people who physically cannot spend as much time as they want virtually together. However, such remote experiences can also tend to lose the feeling of traditional experiences. People lack direct visual presence and no haptic feedback is available. In this paper, we tackle this problem by introducing our system called CheckMate. We combine Augmented Reality and capacitive 3D printed objects that can be sensed on an interactive surface to enable remote interaction while providing the same tangible experience as in co-located scenarios. As a proof-of-concept, we implemented a sample application based on the traditional chess game.
TactileGlove: Assistive Spatial Guidance in 3D Space through Vibrotactile Navigation

Sebastian Günther, Florian Müller, Markus Funk, Jan Kirchner, Niloofar Dezfuli, and Max Mühlhäuser

In Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, Jun 2018

PDF ACM

With the recent advance in computing technology, more and more environments are becoming interactive. For interacting with these environments, traditionally 2D input and output elements are being used. However, recently interaction spaces also expanded to 3D space, which enabled new possibilities but also led to challenges in assisting users with interacting in such a 3D space. Usually, this challenge of communicating 3D positions is solved visually. This paper explores a different approach: spatial guidance through vibrotactile instructions. Therefore, we introduce TactileGlove, a smart glove equipped with vibrotactile actuators for providing spatial guidance in 3D space. We contribute a user study with 15 participants to explore how a different number of actuators and metaphors affect the user performance. As a result, we found that using a Pull metaphor for vibrotactile navigation instructions is preferred by our participants. Further, we found that using a higher number of actuators reduces the target acquisition time than when using a low number.
UPA’18: 3rd International Workshop on Ubiquitous Personal Assistance

Christian Meurisch, Max Mühlhäuser, Philipp M. Scholl, Usman Naeem, Veljko Pejović, Florian Müller, Elena Di Lascio, Pei-Yi Patricia Kuo, Sebastian Kauschke, and Muhammad Awais Azam

In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers - UbiComp ’18, Jun 2018

PDF ACM

Advancements in ubiquitous technologies and artificial intelligence have paved the way for the recent rise of digital personal assistants in everyday life. The Third International Workshop on Ubiquitous Personal Assistance (UPA’18) aims to build on the success of our both previous workshops (namely SmartGuidance), organized in conjunction with UbiComp’16/17, to continue discussing the latest research outcomes of digital personal assistants. We invite the submission of papers within this emerging, interdisciplinary research field of ubiquitous personal assistance that focuses on understanding, design, and development of such digital helpers. We also welcome contributions that investigate human behaviors, underlying recognition, and prediction models; conduct field studies; as well as propose novel HCI techniques to provide personal assistance. All workshop contributions will be published in the supplemental proceedings of the UbiComp’18 conference and included in the ACM Digital Library.
CaMea: Camera-Supported Workpiece Measurement for CNC Milling Machines

Florian Müller, Maximilian Barnikol, Markus Funk, Martin Schmitz, and Max Mühlhäuser

In Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference on - PETRA ’18, Jun 2018

PDF ACM Video

We are experiencing a trend of personal fabrication that allows non-experts to produce highly individualized objects. Beyond 3D printing, this maker movement also approaches larger-scale production machines such as computer numerically controlled (CNC) milling machines that are available in local fabrication laboratories (FabLabs). While the user interfaces and interaction techniques of small-scale 3D printers for household use adapted to the new requirements of non-experts in the last years, such an overhaul of the interfaces for larger machinery is still missing. In this work, we explore the use of augmented reality methods to support novice users in the operation of CNC milling machines. As a first step towards better support for users, we provide a camera-supported graphical and easy-to-use interface for the measurement of raw workpieces inside the machine. In this paper, we contribute our concept CaMea alongside its’ prototype implementation. We further report on the findings of a first early user study.
Personalized User-Carried Single Button Interfaces as Shortcuts for Interacting with Smart Devices

Florian Müller, Martin Schmitz, Markus Funk, Sebastian Günther, Niloofar Dezfuli, and Max Mühlhäuser

In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18, Jun 2018

PDF ACM Video

We are experiencing a trend of integrating computing functionality into more and more common and popular devices. While these so-called smart devices offer many possibilities for automation and personalization of everyday routines, interacting with them and customizing them requires either programming efforts or a smartphone app to control the devices. In this work, we propose and classify Personalized User-Carried Single Button Interfaces as shortcuts for interacting with smart devices. We implement a proof-of-concept of such an interface for a coffee machine. Through an in-the-wild deployment of the coffee machine for approximately three months, we report first initial experiences from 40 participants of using PUCSBIs for interacting with smart devices.
SmartObjects: Sixth Workshop on Interacting with Smart Objects

Florian Müller, Dirk Schnelle-Walka, Tobias Grosse-Puppendahl, Sebastian Günther, Markus Funk, Kris Luyten, Oliver Brdiczka, Niloofar Dezfuli, and Max Mühlhäuser

In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18, Jun 2018

PDF ACM

Smart objects are everyday objects that have computing capabilities and give rise to new ways of interaction with our environment. The increasing number of smart objects in our life shapes how we interact beyond the desktop. In this workshop we explore various aspects of the design, development and deployment of smart objects including how one can interact with smart objects.
An Analysis of Language Impact on Augmented Reality Order Picking Training

Nela Murauer, Florian Müller, Sebastian Günther, Dominik Schön, Nerina Pflanz, and Markus Funk

In Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, Jun 2018

PDF ACM

Order picking is a difficult and cognitively demanding task. Traditionally textual instructions are helping new workers to learn different picking routines. However, the textual instructions are sometimes not written in the workers ’ native languages. In the area of Industry 4.0, where digital functions are finding their way into manufacturing processes, language-independent instructions are possible. Through a user study with 15 participants, we compare textual feedback in the workers’ native language, textual feedback that is written in an unknown foreign language, and visual Augmented Reality (AR) feedback. We found that AR feedback is significantly faster and leads to a lower perceived cognitive load.
Proceedings of the 6th Workshop on Interacting with Smart Objects (SmartObjects)

Jun 2018

PDF Website

These are the proceedings of the 6th Workshop on Interacting with Smart Objects (SmartObjects ’18) in conjunction with CHI’18 held on April 21, 2018, in Montreal, Canada. This volume contains the ten accepted papers. Each submission was reviewed by three program committee members. Objects that we use in our everyday life are ever-expanding their interaction capabilities and provide functionalities that go far beyond their original functionality. They feature computing capabilities and are, thus, able to capture, process and store information and interact with their environments, turning them into smart objects. Their wide range was covered by the submissions to this workshop. Smart objects know something about their users and, thus, allow for natural interaction. Natural interaction, in contrast, does not imply smartness. Smartness requires interaction with users and provides help. There are already commercialized products available that expose their properties and interaction capabilities. To enrich their potential and to lower affordances, they need to communicate to each other. Making sense out of the available data in this field is still an open research question. The overall goal should be to build an interactive ecosystem that (i) seamlessly discovers, connects and talks to its environment, (ii) is ubiquitous and (iii) allows the user to be in control. The workshop examined these issues with regards to the following aspects: ∙ Interactive Experiences ∙ The Future of IoT ∙ Smart Home ∙ AR in the Industry Putting together SmartObjects ‘18 was a team effort. We would like to send out our thanks to everybody who has helped us to organize this event: ∙ The authors, who have written and submitted their papers to the workshop. ∙ The program committee and the external reviewers, for their time and effort to write substantial and constructive review reports. We hope that you will find this program interesting and thought-provoking and that the workshop will provide you with a valuable opportunity to share ideas with other researchers and practitioners from institutions around the world.

2017

BYO*: Utilizing 3D Printed Tangible Tools for Interaction on Interactive Surfaces

Sebastian Günther, Martin Schmitz, Florian Müller, Jan Riemann, and Max Mühlhäuser

In Proceedings of the 2017 ACM Workshop on Interacting with Smart Objects, Mar 2017

PDF ACM

Sharing and manipulating information are essential for collaborative work in meeting scenarios. Nowadays, people tend to bring their own devices as a result of increasing mobility possibilities. However, transferring data from one device to another can be cumbersome and tedious if restrictions like different platforms, form factors or environmental limitations apply. In this paper, we present two concepts to enrich interaction on and between devices through 3D printed customized tangibles: 1) Bring your own information, and 2) bring your own tools. For this, we enable interactivity for low-cost and passive tangible 3D printed objects by adding conductive material and make use of touch-enabled surfaces. Our system allows users to easily share digital contents across various devices and to manipulate them with individually designed tools without additional hardware required. Copyright \copyright 2017 ACM.
Manufacturing Method of Sensor Using 3d Printing and 3d Printer Thereof

Woo Sug Jung, Hwa Suk Kim, Jun Ki Jeon, Seong Kyoun Jo, Hyun Woo Lee, Martin Schmitz, Florian Müller, Andreas Leister, Jan Riemann, Niloofar Dezfuli, Max Mühlhäuser, and Mohammadreza Khalilbeigi

Mar 2017

Website

Disclosed is a manufacturing method of a sensor by using 3D printing and 3D printer therefor. According to an embodiment of the present disclosure , a manufacturing method of a sensor by using 3D printing includes : forming a first shape having an inner space by using a non-conduc tive material , and simultaneously or sequentially , forming an electrode at a preset location in the inner space by using a conductive material ; injecting conductive liquid into the inner space ; and forming a second shape on the first shape by using the non-conductive material to seal the inner space of the first shape .
Cloudbits: Supporting Conversations through Augmented Zero-query Search Visualization

Florian Müller, Sebastian Günther, Azita Hosseini Nejad, Niloofar Dezfuli, Mohammadreza Khalilbeigi, and Max Mühlhäuser

In Proceedings of the 5th Symposium on Spatial User Interaction, Oct 2017

PDF ACM Talk

The retrieval of additional information from public (e.g., map data) or private (e.g., e-mail) information sources using personal smart devices is a common habit in today’s co-located conversations. This behavior of users imposes challenges in two main areas: 1) cognitive focus switching and 2) information sharing. In this paper, we explore a novel approach for conversation support through augmented information bits, allowing users to see and access information right in front of their eyes. To that end, we investigate the requirements for the design of a user interface to support conversations through proactive information retrieval in an exploratory study. Based on the results, we 2) present CloudBits: A set of visualization and interaction techniques to provide mutual awareness and enhance coupling in conversations through augmented zero-query search visualization along with its prototype implementation. Finally, we 3) report the findings of a qualitative evaluation and conclude with guidelines for the design of user interfaces for conversation support.
An Evaluation of Hybrid Stacking on Interactive Tabletops

Jan Riemann, Florian Müller, Sebastian Günther, and Max Mühlhäuser

In Proceedings of the 2017 ACM Workshop on Interacting with Smart Objects, Mar 2017

PDF ACM

Stacking is a common practice of organizing documents in the physical world. With the recent advent of interactive tabletops, physical documents can now coexist with digital documents on the same surface. As a result, systems were developed and studied which allow piling of both types of documents with the physical documents being placed on top of the digital ones. In this paper, we study the concept of true hybrid stacking, allowing users to stack both types of documents in an arbitrary order using a hybrid tabletop system called StackTop. We discuss the results and derive implications for future hybrid tabletop systems with stacking support.
SmartObjects: Fifth Workshop on Interacting with Smart Objects

Dirk Schnelle-Walka, Florian Müller, Tobias Grosse-Puppendahl, Kris Luyten, Max Mühlhäuser, and Oliver Brdiczka

In Proceedings of the 22nd International Conference on Intelligent User Interfaces Companion, Mar 2017

PDF ACM

Smart objects are everyday objects that have computing capabilities and give rise to new ways of interaction with our environment. The increasing number of smart objects in our life shapes how we interact beyond the desktop. In this workshop we explore various aspects of the design, development and deployment of smart objects including how one can interact with smart objects.

2016

ProxiWatch: Enhancing Smartwatch Interaction through Proximity-Based Hand Input

Florian Müller, Sebastian Günther, Niloofar Dezfuli, Mohammadreza Khalilbeigi, and Max Mühlhäuser

In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’16, Mar 2016

PDF ACM

Smartwatches allow ubiquitous and mobile interaction with digital contents. Because of the small screen sizes, tradi-tional interaction techniques are often not applicable. In this work, we show how the degree of freedom offered by the elbow joint, i.e., flexion and extension, can be leveraged as an additional one-handed input modality for smartwatches. By moving the watch towards or away from the body, the user is able to provide input to the smartwatch without a second hand. We present the results of a controlled ex-periment focusing on the human capabilities for proximity-based interaction. Based on the results, we propose guide-lines for designing proximity-based smartwatch interfaces and present ProxiWatch: a one-handed and proximity-based input modality for smartwatches alongside a proto-typical implementation.
FreeTop: Finding Free Spots for Projective Augmentation

Jan Riemann, Mohammadreza Khalilbeigi, Martin Schmitz, Sebastian Doeweling, Florian Müller, and Max Mühlhäuser

In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, May 2016

PDF ACM

Augmenting the physical world using projection technologies or head-worn displays becomes increasingly popular in research and commercial applications. However, a common problem is interference between the physical surface’s texture and the projection. In this paper, we present FreeTop, a combined approach to finding areas suitable for projection, which considers multiple aspects influencing projection quality, like visual texture and physical surface structure. FreeTop can be used in stationary and mobile settings for locating free areas in arbitrary physical settings suitable for projective augmentation and touch interaction.
Liquido: Embedding Liquids into 3D Printed Objects to Sense Tilting and Motion

Martin Schmitz, Andreas Leister, Niloofar Dezfuli, Jan Riemann, Florian Müller, and Max Mühlhäuser

In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, May 2016

PDF ACM

Tilting and motion are widely used as interaction modalities in smart objects such as wearables and smart phones (e.g., to detect posture or shaking). They are often sensed with accelerometers. In this paper, we propose to embed liquids into 3D printed objects while printing to sense various tilting and motion interactions via capacitive sensing. This method reduces the assembly effort after printing and is a low-cost and easy-to-apply way of extending the input capabilities of 3D printed objects. We contribute two liquid sensing patterns and a practical printing process using a standard dual-extrusion 3D printer and commercially available materials. We validate the method by a series of evaluations and provide a set of interactive example applications. \copyright 2016 Authors.
SCWT: A Joint Workshop on Smart Connected and Wearable Things

Dirk Schnelle-Walka, Lior Limonad, Tobias Grosse-Puppendahl, Joel Lanir, Florian Müller, Massimo Mecella, Kris Luyten, Tsvi Kuflik, Oliver Brdiczka, and Max Mühlhäuser

In Companion Publication of the 21st International Conference on Intelligent User Interfaces, Mar 2016

PDF ACM

The increasing number of smart objects in our everyday life shapes how we interact beyond the desktop. In this workshop we discuss how advanced interactions with smart objects in the context of the Internet-of-Thingsshould be designed from various perspectives, such as HCI and AI as well as industry and academia.

2015

A Study of On-body User Interface: PiAM(Palm interAction Module)

Seng-Kyoun Jo, 이현우, Kim Jinsul, Florian Müller, Mohammed Khalilbeigi, and Max Mühlhäuser

Journal of Knowledge Information Technology and Systems, Dec 2015

PDF

User interface providing easy and efficient control environments with user is emerging as a core keyword in ICT market and smart phone industry as the influence and importance of UI has increased recently. To succeed in ICT market, however, UI has to satisfy industrial requirements including simple and easy control, intuitional access and high recognition. Thus study for providing user-friendly interface has been actively researched. In this paper, we investigate user interface based on human’s
A Study on Proximity-based Hand Input for One-handed Mobile Interaction

Florian Müller, Mohammadreza Khalilbeigi, Niloofar Dezfuli, Alireza Sahami Shirazi, Sebastian Günther, and Max Mühlhäuser

In Proceedings of the 3rd ACM Symposium on Spatial User Interaction, Aug 2015

PDF ACM

On-body user interfaces utilize the human’s skin for both sensing input and displaying graphical output. In this paper, we present how the degree of freedom offered by the elbow joint, i.e., exion and extension, can be leveraged to extend the input space of projective user interfaces. The user can move his hand towards or away from himself to browse through a multi-layer information space. We conducted a controlled experiment to investigate how accurately and ef-ficiently users can interact in the space. The results revealed that the accuracy and effciency of proximity-based interactions mainly depend on the traveling distance to the target layer while neither the hand side nor the direction of interaction have a signifcant inuence. Based on our findings, we propose guidelines for designing on-body user interfaces.
Palm-Based Interaction with Head-mounted Displays

Florian Müller, Niloofar Dezfuli, Max Mühlhäuser, Martin Schmitz, and Mohammadreza Khalilbeigi

In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct - MobileHCI ’15, Aug 2015

PDF ACM

Head-mounted displays (HMDs) are an emerging class of wearable devices that allow users to access and alter information right in front of their eyes. However, due to their size and shape, traditional input modalities (e.g., multi-touch sensing on the device) are not practical. In this position paper, we argue that palm-based interactions have a great potential to ease the interaction with HMDs. We outline two interaction concepts and present directions for future research that can lead to more enjoyable and usable interfaces for HMDs.

2012

PalmRC: Imaginary Palm-Based Remote Control for Eyes-Free Television Interaction

Niloofar Dezfuli, Mohammadreza Khalilbeigi, Jochen Huber, Florian Müller, and Max Mühlhäuser

In EuroiTV’12 - Proceedings of the 10th European Conference on Interactive TV and Video, Jul 2012

PDF ACM

User input on television (TV) typically requires a mediator device, such as a handheld remote control. While being a well-established interaction paradigm, a handheld device has serious drawbacks: it can be easily misplaced due to its mobility and in case of a touch screen interface, it also requires additional visual attention. Emerging interaction paradigms like 3D mid-air gestures using novel depth sensors, such as Microsoft’s Kinect, aim at overcoming these limitations, but are known to be e.g. tiring. In this paper, we propose to leverage the palm as an interactive surface for TV remote control. Our contribution is three-fold: (1) we explore the conceptual design space in an exploratory study. (2) Based upon these results, we investigate the effectiveness and accuracy of such an interface in a controlled experiment. And (3), we contribute PalmRC: an eyes-free, palm-surface-based TV remote control, which in turn is evaluated in an early user feedback session. Our results show that the palm has the potential to be leveraged for device-less and eyes-free TV remote interaction without any third-party mediator device. \copyright 2012 ACM.
Leveraging the Palm Surface as an Eyes-Free Tv Remote Control

Niloofar Dezfuli, Mohammadreza Khalilbeigi, Jochen Huber, Florian Müller, and Max Mühlhäuser

In CHI ’12 Extended Abstracts on Human Factors in Computing Systems, May 2012

PDF ACM

User input on television typically requires a mediator device such as a handheld remote control. While being a well-established interaction paradigm, a handheld device has serious drawbacks: it can be easily misplaced due to its mobility and in case of a touch screen interface, it also requires additional visual attention. Emerging interaction paradigms like 3D mid-air gestures using novel depth sensors such as Microsoft’s Kinect aim at overcoming these limitations, but are known for instance to be tiring. In this paper, we propose to leverage the palm as an interactive surface for TV remote control. Our contribution is two-fold: (1) we have explored the conceptual design space in an exploratory study. (2) Based upon these results, we investigated the accuracy and effectiveness of such an interface in a controlled experiment. Our results show that the palm has the potential to be leveraged for device-less and eyes-free TV interactions without any third-party mediator device.
CouchTV: Leveraging the Spatial Information of Viewers for Social Interactive Television Systems - Poster Presentation

Niloofar Dezfuli, Manolis Pavlakis, Florian Müller, Mohammadreza Khalilbeigi, and Max Mühlhäuser

In 10th European Interactive TV Conference 2012, Jul 2012