ConvGuide-VR: a Conversational Guidance Framework for Procedural Learning in Immersive Virtual Reality

Immersive Virtual Reality (IVR) offers highly engaging and interactive environments for procedural learning, yet providing effective guidance remains challenging, as learners may need different levels of help at separate procedural steps. To address this gap, we propose ConvGuide-VR, a conversational guidance framework that leverages speech input and integrates Large Language Models (LLMs) to provide on-demand explanations and context-aware guidance in IVR. ConvGuide-VR aims to enable learners to query the system and receive guidance based on their query, actions, and task state. We implemented the framework in an IVR environment for teaching classical Computer Science procedural knowledge: sorting algorithms. We first evaluated the framework independently and found significant improvements in knowledge acquisition, alongside above-average usability and engagement. We then compared it with a conventional baseline using pre-recorded instructional content, finding that the framework significantly improved task accuracy while achieving comparable learning outcomes, usability, and engagement. Qualitative feedback praises the framework's ability to provide step-by-step guidance and informs future design refinements.