Human-like autonomy emerges from self-play and a pinch of human data

Abstract: Self-play reinforcement learning has recently emerged as a way to train driving policies without any human data. It uses cheap, large-scale simulations to substitute expensive, large-scale human driving demonstrations. A key limitation of this approach is that policies trained through pure self-play can learn effective but alien driving conventions incompatible with people. Previous works attempt to mitigate such behavioral misalignments through extensive reward engineering and domain randomization, which are brittle and labor-intensive. Instead of completely discarding human demonstrations, our method treats them as a regularization objective on top of a minimal safe goal-reaching reward. Like the spice in a good stew, we find that a little human data goes a long way: our method uses only 30 minutes of human demonstrations, 2500× fewer than comparable imitation learning approaches. Resulting policies coordinate with held-out human trajectories and complete training in 15 hours on a single consumer-grade GPU. Videos and full source code are available at https://spiced-self-play.com/.

Introduction. Self-play reinforcement learning (RL) has produced superhuman agents in strategic games [1, 2, 3] and, more recently, has shown promise in real-world domains, such as autonomous driving [4, 5, 6, 7]. The approach elegantly sidesteps a central difficulty in multi-agent learning - how to model the opponent - through the following idea: the agent’s opponent is a copy of itself. The appeal here is that as the agent improves, so does its co-player. This gives rise to an automatically evolving curriculum [8] that takes the policy from random play to skilled behavior entirely through synthetic simulated experience. In zero-sum games, this mechanism, with a sparse measure for success (e.g., +1 when winning a game of chess), is enough to produce strong play against arbitrary opponents. Many real-world settings, however, are not zero-sum. Driving, for instance, can be viewed as a mixed-motive game: each player has individual objectives (reaching a destination safely) but must also coordinate with other road users by adhering to shared norms, expectations, and conventions.

Discussion / Conclusion. We consider a series of experiments aimed at putting the mixing of human driving data with synthetic simulated experience on a more scientific footing. Our central finding is that a small amount of human data, roughly 30 minutes to 3 hours of human driving data, can dramatically move the needle towards human-compatible driving agents. This is three orders of magnitude less than SOTA imitation learning baselines and is achieved without reward engineering or domain randomization techniques. The broader implication is that when simulation is cheap, and some clear metrics for desirable behavior are available, human driving data may be best used not as the primary training signal but as a lightweight anchor that steers policies away from effective-but-alien equilibria. Combining human demonstrations with synthetic simulated experience. Our key finding raises a deeper question that we have only touched the surface of, but is worth exploring further.