Can multi-agent RL handle cooperation without observable signals?

The paper defines "hidden gifts" as "cooperative acts that are not revealed to the recipient" and argues they are a hard case for credit assignment in multi-agent reinforcement learning. It builds a small test, the Manitokan task: agents in a grid world each have an individual door to unlock for an individual reward, and if all of them unlock their doors the group gets a larger collective reward. There is only one key for all the doors, so the collective reward is reachable only if each agent drops the key for the others after using it. Nothing tells an agent that another agent has dropped the key. The result is stated flatly: in the basic version "none of the algorithms tested were able to solve it."

The failed list spans most of the standard families: policy gradient (PG, PPO), meta-learning (SAF), enhanced exploration (MAVEN), counterfactual credit (COMA), and collective value functions (VDN, QMIX, QTRAN, MAPPO). The abstract adds that the failures include "MARL specific architectures" and that the task is "simple." That breadth is the evidence for the claim: no single family of remedies rescues it, and the family built for credit assignment (counterfactual agents) is among those that fail.

The mechanism the paper offers is that the key-drop is "not apparent to the other agents, making it difficult to assign credit between policy updates." The act and its benefit are separated, and the beneficiary sees no trace of it. The paper's human examples make the same point, and they include a parking spot left empty, the last cookie left in the pantry, and the Manitokan caches of the plains Indigenous nations. In each there is "no explicit agreement of a trade or articulation of a 'tit-for-tat'." So the paper is about cooperation that runs on neither agreement nor reciprocity that can be observed.

This sits as a contrast to Can agents learn cooperation by adapting to diverse partners?, where cooperation emerges because agents adapt to what their co-players visibly do. That route needs co-player behavior to be legible in context, and hidden gifts remove that legibility by construction. The algorithms listed in this excerpt do not include sequence-model agents trained against diverse co-players, so the excerpt neither shows nor suggests that such agents would fail; it marks a condition, an act with no observable trace, under which the mutual-shaping story has nothing to work with. There is a looser kinship with Can success feedback teach agents to skip required steps?: there the reward channel teaches the wrong lesson, here it carries no usable lesson about the act at all.

The excerpt is silent on the size of the failure (returns, success rates, seeds, training budget), on the grid size and number of agents, and on what other versions of the task exist beyond the "basic version," so it does not say whether any variant or algorithm succeeds. The credit-assignment explanation is the authors' account of why the algorithms fail; the excerpt reports no test that isolates it. The excerpt also says nothing about LLM agents. What it supports is narrower: a cooperative act that leaves no trace for its beneficiary can defeat every one of nine common MARL algorithms in a simple environment, so that failure cannot be put down to task difficulty alone.