A score tells you whether an AI did well, but not which way it should change — so two very different failures look identical.
Why do scalar evaluation scores collapse distinguishable agent behaviors?
This explores why reducing an agent's behavior to a single number — a reward or eval score — erases the differences between distinct strategies, and what information gets lost in that compression.
This explores why collapsing an agent's behavior into a single scalar score — a reward, a benchmark number — erases the very distinctions between strategies you'd want to see. The corpus's sharpest answer is that a scalar is the wrong shape for the signal. Natural feedback actually carries two orthogonal kinds of information: evaluative (how well an action did) and directive (which way it should change). A scalar keeps the first and throws away the second, so two agents that failed for completely different reasons — one that overshot, one that never explored — get stamped with the same number and become indistinguishable Can scalar rewards capture all the information in agent feedback?. The distinguishing information wasn't unmeasurable; it was discarded by the reduction.
The collapse isn't only in how we read behavior — training on scalars actively produces it. Reinforcement learning optimized against a single reward drives policies to converge on a narrow band of reward-maximizing strategies, the same entropy-collapse dynamic seen in reasoning models: diversity that existed in the base model gets squeezed out because the objective rewards only the winning move, not the range of viable ones Does reinforcement learning squeeze exploration diversity in search agents?. So a scalar both hides distinguishable behaviors at evaluation time and, when used as a training target, homogenizes the behaviors themselves.
What's striking is how much richer feedback becomes the moment you refuse the scalar. Reflexion shows agents turning a bare success/failure bit into written self-diagnoses — but notice the binary signal isn't the useful part; the useful part is the verbal reflection the agent writes around it, which preserves the "why" a number would flatten Can agents learn from failure without updating their weights?. SkillRL goes further, treating successes and failures as different kinds of information entirely — successes as concrete demonstrations, failures as abstracted lessons — an asymmetry a single score literally cannot represent, and one that outperforms uniform treatment Should successful and failed episodes be processed differently?.
There's a cross-domain echo worth pulling out: the same flattening that hides bad behaviors also hides good discriminative machinery. DyLAN scores each agent's contribution to a team not with one number but through propagation and aggregation across the interaction, recovering per-agent signal that a team-level success rate would blur into a single indistinguishable blob Can multi-agent teams automatically remove their weakest members?. And omniscient social simulations look competent precisely because a top-line score hides the grounding work models skip — the failures only become distinguishable once you stop measuring at the aggregate Why do LLMs fail when simulating agents with private information?.
The thread across all of these: a scalar is a projection, and projection is lossy by construction. It answers "how much" while destroying "which way" and "why" — and those two are exactly the axes on which agent behaviors differ. The fix isn't a better number; it's keeping feedback in a shape (directive signals, verbal reflections, asymmetric success/failure handling, per-agent contribution) that has room for the distinctions you care about.
Sources 6 notes
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
DyLAN's three-step importance scoring mechanism (propagation, aggregation, selection) quantifies individual agent contributions and automatically removes uninformative agents during inference, optimizing team composition without task-specific tuning.
Show all 6 sources
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Useful Memories Become Faulty When Continuously Updated by LLMs
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
- Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Real-Time Procedural Learning From Experience for AI Agents
- Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trust