This paper introduces an augmented reality (AR) captioning framework designed
to support Deaf and Hard of Hearing (DHH) learners in STEM classrooms by
integrating non-verbal emotional cues into live transcriptions. Unlike
conventional captioning systems that offer only plain text, our system fuses
real-time speech recognition with affective and visual signal interpretation,
including facial movements, gestures, and vocal tone, to produce emotionally
enriched captions. These enhanced captions are rendered in an AR interface
developed with Unity and provide contextual annotations such as speaker tone
markers (per esempio., “concerned”) and gesture indicators (per esempio., “nods”). The system
leverages live camera and microphone input, processed through AI models to
detect multimodal cues. Findings from preliminary evaluations suggest that this
AR-based captioning approach significantly enhances comprehension and reduces
cognitive effort compared to standard captions. Our work emphasizes the
potential of immersive environments for inclusive, emotion-aware educational
accessibility.

Questo articolo esplora i giri e le loro implicazioni.

Scarica PDF: