Modeling the Quality of Dialogical Explanations

Abstract Explanations are pervasive in our lives. Mostly, they occur in dialogical form where an explainer discusses a concept or phenomenon of interest with an explainee. Leaving the explainee with a clear understanding is not straightforward due to the knowledge gap between the two participants. Previous research looked at the interaction of explanation moves, dialogue acts, and topics in successful dialogues with expert explainers. However, daily-life explanations often fail, raising the question of what makes a dialogue successful. In this work, we study explanation dialogues in terms of the interactions between the explainer and explainee and how they correlate with the quality of explanations in terms of a successful understanding on the explainee’s side. In particular, we first construct a corpus of 399 dialogues from the Reddit forum Explain Like I am Five and annotate it for interaction flows and explanation quality. We then analyze the interaction flows, comparing them to those appearing in expert dialogues. Finally, we encode the interaction flows using two language models that can handle long inputs, and we provide empirical evidence for the effectiveness boost gained through the encoding in predicting the success of explanation dialogues.

Introduction. Explanations play a significant role in our daily life. Typically, they are realized through dialogues, where one person is an explainer while the other takes the explainee position. The explainer’s primary goal is to convey information about a particular concept or phenomenon to the explainee clearly and concisely. However, ensuring that the explainee understands an explanation successfully is challenging: Effective explanations require more than just information delivery. Expert explainers usually plan an explanation strategy by choosing appropriate explanation moves, dialogue acts, and topics to ensure optimal comprehension on the explainee side (Wachsmuth and Alshomary, 2022). Additionally, explainees may actively engage in dialogues by asking clarification questions and providing feedback to ensure they understand the information correctly (Madumal et al., 2019). Most previous research has studied monological explanations (Fan et al., 2019; Situ et al., 2021), where an explainer provides a single-turn explanation, ignoring the role of the explainee in the interaction.

Discussion / Conclusion. We studied real-life explanation dialogues and how to assess their success. To this end, we constructed a dataset of real-life explanation dialogues from the Explain Like I am Five Subreddit. We annotated it according to the explanation taxonomy of Wachsmuth and Alshomary (2022) and rated the quality of these dialogues in terms of the explainee’s understanding. Our analysis provides insights into the difference between these dialogues and expert explanation dialogues. We then assessed the performance of pre-trained language models in predicting the quality of explanation dialogues and found that encoding specific interaction flows into their input boosts effectiveness. In quantifying the explanation dialogue quality, we relied on the annotators’ intuition of guessing the explainee’s understanding.