What we did:We designed ShadAR, an Augmented Reality application that enables real-time transformation of visual perception through shader generation using large language models.
What we found:We demonstrate that ShadAR allows users to express their visual intent in natural language, which is then translated into shader code for immediate application in Augmented Reality experiences.
Takeaway:Our application highlights the potential of using large language models for enhancing accessibility and creativity in visual experiences.
Abstract
Augmented Reality (AR) can simulate various visual perceptions, such as how individuals with colorblindness see the world. However, these simulations require developers to predefine each visual effect, limiting flexibility. We present ShadAR, an AR application enabling real-time transformation of visual perception through shader generation using large language models (LLMs). ShadAR allows users to express their visual intent via natural language, which is interpreted by an LLM to generate corresponding shader code. This shader is then compiled real-time to modify the AR headset’s viewport. We present our LLM-driven shader generation pipeline and demonstrate its ability to transform visual perception for inclusiveness and creativity.
We explored how AR memes influence face-to-face conversations through a survey of meme users and a multi-user prototype study in which pairs of friends designed and used their own AR meme visualizations.
We conducted a user study comparing three order picking assistance systems: textual instructions in German, textual instructions in Finnish, and an Augmented Reality system using a Microsoft HoloLens.
