Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy

Abstract: Educational chatbots come with a promise of interactive and personalized learning experiences, yet their development has been limited by the restricted free interaction capabilities of available platforms and the difficulty of encoding knowledge in a suitable format. Recent advances in language learning models with zero-shot learning capabilities, such as ChatGPT, suggest a new possibility for developing educational chatbots using a prompt-based approach. We present a case study with a simple system that enables mixed-turn chatbot interactions and we discuss the insights and preliminary guidelines obtained from initial tests. We examine ChatGPT's ability to pursue multiple interconnected learning objectives, adapt the educational activity to users' characteristics, such as culture, age, and level of education, and its ability to use diverse educational strategies and conversational styles. Although the results are encouraging, challenges are posed by the limited history maintained for the conversation and the highly structured form of responses by ChatGPT, as well as their variability, which can lead to an unexpected switch of the chatbot's role from a teacher to a therapist.

Introduction. Chatbots are computer programs designed to simulate conversation with human users. Educational chatbots, in particular, have gained increasing attention in recent years for their potential to provide interactive and personalized educational activities. They have been applied in different educational domains and designed for different roles with different interaction styles (Kuhail et al., 2023). It has also been proposed that they can help address educational inequalities in areas plagued by poverty and limited access to quality education. However, their development has been mostly focused on chatbot-driven conversation flow due to the complexity of mixed-initiative and user-driven approaches, the restricted interaction capabilities of available platforms, the lack of adequate training sets (Pérez et al., 2020), and the substantial efforts required to encode the necessary knowledge in a suitable format, notwithstanding the wide availability of topic-specific information.

Discussion / Conclusion. Our study showed that ChatGPT with the correct prompt was effective in playing the role of the teacher in interactive educational conversations covering multiple learning objectives related to social media literacy. The chatbot was also able to tailor educational activities to different users’ characteristics, such as their cultural background and age group. While searching for effective prompts we repeatedly observed low-quality responses that did not match the requests in the prompt, reflecting a degree of limited understanding (Borji, 2023; Mahowald et al., 2023; Webson & Pavlick, 2022, Valmeekam et al., 2022, Zhang et al., 2022). The observed stereotyped responses and lack of flexibility during educational conversations may be in part due to the limited relative amount of conversational educational samples in the original training dataset of GPT3 as well as a refinement phase, that comprises the Reinforcement Learning through Human Feedback, that we suspect was tuned to generate textbook like answers, i.e. question answering behaviors.