Improving the Effectiveness of Chatbots by Incorporating Self-Assessed User Knowledge into the Question-Answering Process

Chatbots, personal assistants, and large language models (LLMs) have become pervasive in our world. A major function they serve is to provide information and answer questions. Collectively, these technologies typically have two major weaknesses: they provide general answers to questions they are asked without regard to the specific knowledge needs of the user and they do not assess whether the user actually understood the answers or information provided. The present paper addresses the first weakness by creating a self-assessment chatbot that helps a user to self-assess what s/he knows about a topic and then uses that knowledge when answering the user’s questions with an eye toward filling in the knowledge gaps. This self-assessment chatbot was presented to college students learning math and compared to Chat GPT, which simply answers questions without such self-assessment information. Results showed that students using a self-assessment chatbot scored 10 points or the equivalent of a full letter grade higher on a posttest than those using the standard Chat GPT.