Can simulated APIs and token-level credit assignment train better tool-using agents?

Agentic RL training that involves tool use faces two operational problems that compound each other. First, training agents through interaction with real-world APIs is expensive and unstable — APIs rate-limit, fail intermittently, change over time, and cost money per call. RL training requires many thousands of trajectories, so even small per-call frictions accumulate into significant training instability and expense. Second, when outcome rewards are sparse (final task success or failure), they cannot reliably credit the specific tool calls that contributed to success — the same trajectory may have correct tool calls mixed with incorrect ones, and the outcome reward provides no signal to distinguish them.

DeepAgent's ToolPO addresses both problems together. The real-API problem is replaced with LLM-simulated APIs — a separate model approximates the behavior of the tools the agent would call, providing the interaction signal without the cost or instability of live calls. This is not new in principle (simulators have been used in RL for decades) but the LLM-simulator construction is well-suited to the tool-call setting because the APIs the agent interacts with are themselves often natural-language-shaped (search results, knowledge bases, structured queries).

The sparse-reward problem is addressed by tool-call advantage attribution. Rather than backpropagating outcome rewards uniformly across the trajectory, ToolPO attributes advantage specifically to the tokens that constitute tool invocations. A correct tool call in a trajectory that ultimately succeeds gets positive credit; a correct tool call in a trajectory that ultimately fails (because of a later mistake) still gets the credit it deserves; an incorrect tool call gets penalized even when the trajectory succeeds despite it.

The combined effect is more stable and more sample-efficient agentic RL training. The training loop runs against the simulator (stability), and the gradient signal targets the right tokens (efficiency). For tool-using agent deployments where direct RL on production APIs is impractical, this combination is a viable training architecture.