Can AI agents learn to help each other when the good deed is invisible and the payoff goes to someone else?
Can agents learn cooperation from reward signals alone without seeing helpers?
This explores whether agents can learn to cooperate when the only feedback is a reward score and they can't see who helped them, meaning the helpful act is hidden and its payoff arrives later, to someone else.
This explores whether agents can learn to cooperate when reward is the only feedback and the helpful acts are invisible to them. The corpus's most direct answer is no, not with today's standard methods. In the Manitokan task, agents must secretly leave a shared key so that others can succeed. Every algorithm tested failed, including policy gradients, meta-learning, and counterfactual credit assignment Can multi-agent RL handle cooperation without observable signals?. The helper pays a cost now, someone else collects the payoff later, and nothing in the reward links the two. Even methods built to ask 'what did my action contribute?' can't bridge that gap.
Cooperation does emerge elsewhere in the corpus, but in settings where the partner's behavior is visible. Sequence-model agents trained against a diverse set of co-players learn to adapt in-context to whoever they're facing. Cooperation appears because each side is vulnerable to being exploited by the other, so both shift toward cooperating, with no hardcoded assumptions about partners Can agents learn cooperation by adapting to diverse partners?. Gemini-based agents reach stable cooperation in social dilemmas designed to block the usual routes. They do it by reasoning that a partner probably decides the way they do Can AI agents cooperate without explicit incentives or enforcement?. Both approaches rely on the agent being able to read its partner's behavior. Neither is tested on hidden gifts.
So the pattern is that reward alone can teach cooperation when the reward is tangled up with a partner's observable conduct. It breaks when the helpful act leaves no trace. Two other notes suggest why the hidden case is so hard, and where a fix might come from. Scalar rewards capture how well something went but discard how behavior should change Can scalar rewards capture all the information in agent feedback?. In a hidden-gift task, the reward can say the group did well but can't point to the secret act that caused it.
One possible workaround is to build the missing signal internally. ΔBelief-RL gives an agent dense, per-turn credit from how its own beliefs about the answer shift, with no critic network Can an agent's own beliefs guide credit assignment without critics?. It was tested only on single-agent 20 Questions. Whether the same trick could let an agent notice 'something just helped me' and trace it back to an unseen partner is untested in this collection.
Sources 5 notes
In the Manitokan task, where agents must secretly leave a shared key for others to succeed collectively, every tested algorithm failed—spanning policy gradients, meta-learning, and counterfactual credit assignment methods. The core problem is that unobserved helpful acts create credit-assignment gaps that current approaches cannot bridge.
Sequence model agents trained against diverse co-players develop in-context best-response strategies that naturally resolve into cooperation. Mutual vulnerability to exploitation creates pressure that drives cooperative mutual adaptation without hardcoded assumptions or timescale separation.
Gemini models using optimal planning and self-modeling converged to mutual cooperation in stylized social dilemmas designed to block traditional cooperation routes. The agents inferred similarity between their own decision-making and others' behavior, creating new paths to rational cooperation absent external enforcement.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Multi-agent cooperation through in-context co-player inference
- A game theory for foundation models shows new paths to rational cooperation through similarity inference
- Reward Reasoning Model
- Reinforcement Learning via Self-Distillation
- Drop the Hierarchy and Roles: How Self-Organizing LLM Agents Outperform Designed Structures
- Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs
- Intrinsic Credit Assignment for Long Horizon Interaction
- The challenge of hidden gifts in multi-agent reinforcement learning