Can human data steer self-play RL toward human-compatible behavior?

Pure self-play RL sidesteps the hardest problem in multi-agent learning — how to model the opponent — by making the opponent a copy of the agent, which yields an automatically evolving curriculum from random to skilled play. But this elegance has a failure mode in mixed-motive settings like driving: the policy converges to conventions that are effective yet alien, incompatible with how humans actually coordinate. Prior fixes leaned on brittle, labor-intensive reward engineering and domain randomization to paper over the misalignment.

The SPICED result reframes the role of human data entirely. Instead of using demonstrations as the primary training signal (imitation learning) or discarding them (pure self-play), it treats them as a regularization objective layered on top of a minimal safe goal-reaching reward. Thirty minutes of human driving — roughly 2500x less than comparable imitation baselines — is enough to pull the policy toward human-compatible equilibria, training in 15 hours on one consumer GPU. The mechanism matters: the self-play reward supplies competence, the human anchor supplies convention-compatibility, and these are separable objectives.

This generalizes a principle that recurs across RL post-training: when synthetic experience is cheap and clear success metrics exist, a small amount of human data is better spent shaping which equilibrium the policy selects than on teaching the task itself. It parallels how a little demonstration data can align behavior implicit in a base model without expensive preference labels — the human signal amplifies and steers rather than instructs from scratch.