ShadAR: LLM-driven Shader Generation to Transform Visual Perception in Augmented Reality
Mei, Yanni and Wendt, Samuel and Müller, Florian and Gugenheimer, Jan
Abstract: Augmented Reality (AR) can simulate various visual perceptions, such as how individuals with colorblindness see the world. However, these simulations require developers to predefine each visual effect, limiting flexibility. We present ShadAR, an AR application enabling real-time transformation of visual perception through shader generation using large language models (LLMs). ShadAR allows users to express their visual intent via natural language, which is interpreted by an LLM to generate corresponding shader code. This shader is then compiled real-time to modify the AR headset’s viewport. We present our LLM-driven shader generation pipeline and demonstrate its ability to transform visual perception for inclusiveness and creativity.
@inproceedings{meiShadARLLMdrivenShader2025,title={{{ShadAR}}: {{LLM-driven}} Shader Generation to Transform Visual Perception in {{Augmented Reality}}},shorttitle={{{ShadAR}}},booktitle={2025 {{IEEE International Symposium}} on {{Mixed}} and {{Augmented Reality Adjunct}} ({{ISMAR-Adjunct}})},author={Mei, Yanni and Wendt, Samuel and M{\"u}ller, Florian and Gugenheimer, Jan},year={2025},month=oct,pages={959--960},issn={2771-1110},doi={10.1109/ISMAR-Adjunct68609.2025.00267},urldate={2026-01-05},keywords={Augmented reality,Augmented Reality,Codes,Creativity,Natural languages,Pipelines,Real-time systems,Shader Generation,Systems architecture,Transforms,Visual effects,Visual perception,Visual Perception Transformation},}
