Do clarifying dialogues help LLMs answer better than rewritten inputs?

The excerpt reports that, when comparing ways of resolving an ambiguous input, "systems tend to perform better with clarifying questions and answers than with disambiguated inputs, particularly for QA and NLI." Both conditions give the model equivalent information about the user's intended interpretation — one as a clarifying exchange, the other as a rewritten, disambiguated version of the original input — yet the clarifying-dialogue condition produces more accurate downstream responses.

The paper attributes the gap to how the disambiguated inputs were constructed: "these datasets create disambiguated revisions of each ambiguous input by applying a minimal set of token-level edits to the initial input. While this makes disambiguations easier to annotate and compare, it comes at the cost of the naturalness of the resulting disambiguations." Clarifying questions and answers, by contrast, "do not have the same minimal-edit constraints and more closely resemble these systems' pretraining distribution" — the model has seen far more natural clarifying exchanges in training than it has seen minimally-edited paraphrases of ambiguous statements.

This complicates the premise of Do standard NLP benchmarks hide LLM ambiguity failures?: not only do ambiguity benchmarks exclude hard cases, the minimal-edit method used to construct "disambiguated" ground truth in such benchmarks produces text the model handles worse than natural dialogue would. It also extends Can command generation replace intent classification in dialogue systems?: both findings point toward dialogue-native formats outperforming formats that compress interaction into a single rewritten or classified input. And it adds a mechanism to Which clarifying questions actually improve user satisfaction? — that note shows question framing affects user satisfaction, while this finding shows the clarifying-exchange format itself, independent of phrasing, improves the model's own answer accuracy.

The comparison is reported for QA and NLI specifically ("particularly for QA and NLI"), with the excerpt not stating whether the same gap held for machine translation; the clarifying questions and answers were themselves generated by an oracle system via few-shot prompting of GPT-3.5, not collected from real users, so the result speaks to model behavior under scripted, oracle-quality clarification rather than to live clarifying dialogue with people. The excerpt does not report the size of the accuracy gap.