Design and Evaluation of a Conversational Agent for Formative Assessment in Higher Education

In recent years, there have been attempts to design and use conversational agents for educational assessments (i.e., conversation-based assessments: CBA).To address the limited research on CBA, we designed a CBA to serve as a formative assessment of higher-education students' knowledge and scaffold their learning by providing support and feedback.CBA was designed using Rasaan artificial intelligence-based tooland shared with students via Google Chat.The conversation data showed that CBA produced high standard accuracy measures and confidence scores.The findings suggest that ensuring the accuracy of CBA with constructed-response items is more challenging than CBA with selected-response items.In addition, a cognitive walkthrough of CBA provided preliminary evidence for the use of CBA as an interactive assessment tool.According to survey responses, most of the participating students reported positive attitudes toward CBA and its use to improve their assessment experience and learning.