Do RL agents accidentally use environments as memory?
Explores whether reinforcement learning agents unintentionally create external memory through environmental artifacts—like trails and marks—without being explicitly trained to do so, and whether this constitutes genuine cognitive extension.
"Artifacts as Memory Beyond the Agent Boundary" (2604.08756) formalizes how the environment can functionally serve as an agent's memory within reinforcement learning. The key contribution is a mathematical proof: certain observations — called artifacts — reduce the information needed to represent history (Theorem 1). An artifact is an observation that reliably informs about past events. A folded page corner tells you where you stopped reading without remembering the page number.
The striking finding: external memory arises unintentionally. RL agents given standard navigation objectives (sparse reward for reaching a goal) develop path-following behavior by reading and writing information to the environment without any explicit objective directing them to do so. In dynamic path environments, agents record traces of previous interactions that go on to guide future behavior — this emerges naturally from credit assignment in sufficiently complex environments.
Three criteria from situated cognition (Michaelian 2012, Sims & Kiverstein 2022) are shown to hold: (1) Survival relevant — agents in artifactual environments consistently accumulate more reward. (2) Susceptible to change — artifacts in dynamic environments encode/store/retrieve information. (3) Selection — through repeated credit assignment, policies bias navigation toward goal-relevant locations.
This directly grounds the Extended Mind thesis (Clark & Chalmers 1998) in computational experiment. Since Did Chalmers abandon his own Extended Mind principles?, the Artifacts paper demonstrates that Clark's original insight — cognitive processes extend into the environment — holds empirically for artificial agents, not just philosophical thought experiments.
The implication for agent design is provocative: rather than scaling internal parameters, performance gains may arise from environments that coevolve with the agent. Current architectures may already suffice for competent performance but require appropriate environmental scaffolding. This challenges the scaling paradigm while connecting to Can agents learn continuously from experience without updating weights? — AgentFly stores experiences in an explicit case bank, while the Artifacts work shows that even implicit environmental traces can serve as memory without any designed storage mechanism.
Inquiring lines that read this note 29
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How should agents balance memory condensation to optimize context efficiency?- Can environmental scaffolding replace internal memory scaling in agent design?
- Can episodic memory of UI traces improve open-world agent adaptation?
- How does credit assignment drive agents to write information into environments?
- What makes exploration and reflection rewards verifiable in agentic environments?
- What counts as genuine memory under the Extended Mind thesis?
- Can episodic memory alone enable learning without parameter updates?
- How can memory shift from a passive datastore to an actively trained component?
- How does continuous implicit memory formation differ from explicit memory encoding?
- What happens when agents interact with environments and learn from their own mistakes?
- Can simulation fidelity limit what agents learn from trained world models?
- How do world models create indirect causal grounding without physical environment contact?
- Can external summarization solve exploration problems in complex real-world environments?
- What details do high-level trajectory abstractions lose that state-grounded recall preserves?
- How do trajectory quality and memory hygiene differ as evaluation metrics?
- Does environment stochasticity force models to generalize better across trajectory variations?
- What determines whether accumulated state generalizes spuriously across continual learning domains?
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Artifacts as Memory Beyond the Agent Boundary
- Rethinking Memory as Continuously Evolving Connectivity
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Large Language Model Agents Are Not Always Faithful Self-Evolvers
Original note title
RL agents unintentionally use spatial environments as external memory — artifacts that inform the past reduce the information needed to represent history without explicit memory objectives