This paper introduces an augmented reality (AR) captioning framework designed
to support Deaf and Hard of Hearing (DHH) learners in STEM classrooms by
integrating non-verbal emotional cues into live transcriptions. Unlike
conventional captioning systems that offer only plain text, our system fuses
real-time speech recognition with affective and visual signal interpretation,
including facial movements, gestures, and vocal tone, to produce emotionally
enriched captions. These enhanced captions are rendered in an AR interface
developed with Unity and provide contextual annotations such as speaker tone
markers (e.g., “concerned”) and gesture indicators (e.g., “nods”). The system
leverages live camera and microphone input, processed through AI models to
detect multimodal cues. Findings from preliminary evaluations suggest that this
AR-based captioning approach significantly enhances comprehension and reduces
cognitive effort compared to standard captions. Our work emphasizes the
potential of immersive environments for inclusive, emotion-aware educational
accessibility.

Este artículo explora los viajes en el tiempo y sus implicaciones.

Descargar PDF: