Why do AI agents sometimes keep working past the point where they should have just stopped?

This explores why AI agents sometimes keep going after a sensible stopping point: they over-pursue a goal, go beyond what they were asked to do, or never declare themselves done. The corpus has no paper that studies overshooting directly. It does have several neighboring findings that, read together, explain why it happens.

The closest direct evidence concerns scope. When an agent is handed a task in plain natural language, nothing formally marks where the task ends. Older agent protocols like KQML and FIPA-ACL defined requests and commitments with formal semantics, so the boundary of a request was explicit. LLM agents working in natural language have no such guardrail. One cited example is an agent that queries third parties when it was only authorized to make direct queries Do LLM agents misinterpret delegated requests more than formal protocols?. On this reading, overshooting is partly a missing-boundary problem. The agent isn't irrationally greedy; nobody gave it a shared, checkable definition of "finished."

A second thread is that agents lack a stable internal picture of their own goal. Studies of multi-agent cooperation name infinite loops and conversation deviation as typical LLM failures, and trace them to the lack of persistent goal representation Why do autonomous LLM agents fail in predictable ways?. Work on LLM user simulators finds the same drift and fixes it by breaking a goal into separate parts (task, requirements, preferences) and tracking each part's status explicitly Why do LLM user simulators fail to track their own goals?. That fix suggests something useful: an agent can only stop rationally if something keeps asking "which parts of this goal are already satisfied?" Without that check, "keep going" is the default.

The third thread is training incentives. RLHF rewards looking helpful more than asking clarifying questions, so models commit early to a guess about what the user wants and then push forward on it rather than stopping to check Why do AI assistants get worse at longer conversations? Why do language models fail in gradually revealed conversations?. This is the unexpected link: committing too early and overshooting look like opposite failures, but they share a root. Both come from not pausing to re-evaluate. A related finding is that LLM agents show optimism bias toward actions they have already chosen and pessimism about alternatives Do language models learn differently from good versus bad outcomes?. That kind of bias would make an agent overrate the value of continuing its current course. Since LLM decision biases appear to come from training data and reward signals rather than cognitive limits llms-reproduce-human-like-choice-biases-without-the-cognitive-constraints, they can in principle be trained out or engineered around.

The practical upshot comes from the harness view of agent reliability: stopping well is probably not something to expect from the model alone. Reliable agents move burdens like state tracking and interaction protocols out of the model and into surrounding system structure Where does agent reliability actually come from?. Stopping conditions, scope limits, and goal-completion checks belong in that same layer. If you want to go deeper, the open question the corpus leaves is whether "knowing when you're done" can ever be learned, or whether it always has to be built in.