Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset

This work proposes a new benchmark for empathetic dialogue generation and EMPATHETICDIALOGUES, a novel dataset of 25k conversations grounded in emotional situations. Our experiments indicate that dialogue models that use our dataset are perceived to be more empathetic by human evaluators, compared to models merely trained on large-scale Internet conversation data.

Each dialogue is grounded in a specific situation where a speaker was feeling a given emotion, with a listener responding (Figure 2). The new resource consists of crowdsourced one-on-one conversations, and covers a large set of emotions in a balanced way. This dataset is larger and contains a more extensive set of emotions than many similar emotion prediction datasets from other text domains

| | | | |---|---|---| |Surprised|got,shocked,really|that's,good,nice| |Excited|going,wait,i'm|that's,fun,like| |Angry|mad,someone,got|oh,would,that's| |Proud|got,happy,really|that's,great,good| |Sad|really,away,|get sorry,oh,hear| |Annoyed|get,work,really|that's,oh,get| |Grateful|really,thankful,i'm|that's,good,nice| |Lonely|alone,friends,i'm i'|m,sorry,that's| |Afraid|scared,i'm,night|oh,scary,that's| |Terrified|scared,night,i'm|oh,that's,would| |Guilty|bad,feel,felt|oh,that's,feel| |Impressed|really,good,got|that's,good,like| |Disgusted|gross,really,saw|oh,that's,would| |Hopeful|i'm,get,really|hope,good,that's| |Confident|going,i'm,really|good,that's,great| |Furious|mad,car,someone|oh,that's,get| |Anxious|i'm,nervous,|going oh,good,hope| |Anticipating|wait,i'm,|going sounds,good,hope| |Joyful|happy,got,i'm|that's,good,great| |Nostalgic|old,back,|really good,like,time| |Disappointed|get,really,work|oh,that's,sorry| |Prepared|ready,i'm,going|good,that's,like| |Jealous|friend,got,get|get,that's,oh| |Content|i'm,life,happy|good,that's,great| |Devastated|got,really,|sad sorry,oh,hear| |Embarrassed|day,work,got oh,|that's,i'm| |Caring|care,really,taking|that's,good,nice| |Sentimental|old,really,time|that's,oh,like| |Trusting|friend,trust,know|good,that's,like| |Ashamed|feel,bad,felt oh,|that's,i'm| |Apprehensive|i'm,nervous,|really oh,good,well| |Faithful|i'm,would,years|good,that's,like|

As a result, we developed the Empathetic Question Taxonomy (EQT) with two distinguished branches: question acts describe semantic-driven features of questions (e.g., ask for confirmation, positive rhetoric), whereas question intents characterize their emotion-regulation functions targeted at the interlocutor’s emotional state (e.g., sympathize, amplify excitement). As it will be revealed further (§7), an empathetic listener can use different question acts to deliver the same intent, justifying the proposed branching.

Question acts Request information (38.7%): Ask for new factual information

Ask about consequence (21.0%): Ask about the result of the described action or situation

Ask about antecedent (17.1%): Ask about the reason or cause of the described state or event

Suggest a solution (8.7%): Provide a specific solution to a problem in a form of a question

Ask for confirmation (5.8%): Ask a question to confirm or verify the listener’s understanding of something that has been described by the speaker

Suggest a reason (5.2%): Suggest a specific reason or cause of the event or state described by the speaker in a form of a question

Irony (1.3%): Ask a question that suggests the opposite of what the speaker may expect, usually to be humorous or pass judgement