A chatbot trained hard to ace academic quizzes can still feel awkward and robotic when you just try to talk to it.

This explores why tuning on FLAN v2's academic-style tasks raises MMLU scores but doesn't produce a good conversational assistant. The corpus has no note on FLAN v2 itself, so it can't confirm the specific benchmark gap. What it does have is a strong body of work on the underlying split: being good at answering a well-specified question in one turn is a different skill from holding a conversation. Briefly, from outside the corpus: FLAN v2 is mostly short, single-turn tasks with templated prompts and terse target answers. That's close to the format MMLU tests, and far from what a chat user expects.

The clearest evidence for the split is the multi-turn degradation research. Models that score around 90% when a task arrives in one complete message drop to about 65% when the same information comes in gradually over a natural conversation Why do AI assistants get worse at longer conversations?. Across more than 200,000 conversations, every major model lost about 39% on average. The cause was that models lock into early guesses and can't recover from them Why do language models fail in gradually revealed conversations?. One analysis argues this isn't a loss of capability at all. The model still knows the answer, but it has misread what the user wants Why do language models lose performance in longer conversations?. A benchmark like MMLU never tests for that kind of failure, because it hands over the whole question at once.

A second idea fits FLAN-style training even more directly: models only learn the conversational behaviors their training data actually contains. One study found that even top models get pulled off topic by distracting conversational turns. Fine-tuning on just 1,080 synthetic dialogues that included distractors fixed much of the problem, which suggests the gap was missing training signal, not limited model capacity Why do language models engage with conversational distractors?. In that study's words, models learn 'what to do' instructions but not 'what to ignore' instructions. A collection of single-turn academic tasks contains neither topic drift, nor clarifying questions, nor follow-ups, so a model tuned on it has never practiced them.

The gap also shows up as passivity. Conversational agents tend to wait for queries rather than steer a conversation, plan, or take initiative, because their training rewards responding, not leading Why can't conversational AI agents take the initiative?. A related finding: models handle discourse relations well when explicit connectives like 'because' or 'however' are present, but drop to about 25% accuracy when the link is only implied Why does ChatGPT fail at implicit discourse relations?. Real conversation runs on implied connections like these. Templated task prompts mostly spell them out.

The broader point is that a knowledge benchmark and a chat evaluation measure two separate abilities, and a training mix can push one up without touching the other. This is also why the field moved toward training on synthetic or simulated conversations. Work on LLM user simulators shows that generated dialogues can be made realistic enough to train on by conditioning on a user profile and on each turn's intent Can controlled latent variables make LLM user simulators realistic?. That supplies the multi-turn, intent-shifting signal that FLAN-style data lacks.