Can AI agents that click through apps like a person learn to cope with real software by practicing live?
What role does online RL play in scaling GUI agents?
This explores whether letting GUI agents learn from live trial and error (online reinforcement learning) is what lets them scale, and how it fits with other ways of improving them.
This explores whether letting GUI agents learn from live trial and error (online reinforcement learning) is what lets them scale, and how it fits with other ways of improving them. The corpus has only one note squarely on GUI agents, and it treats online RL as one essential ingredient, not the whole recipe. Why do GUI agents fail when leaving the lab? says agents tuned on simulated benchmarks stumble in real use. Fixing that takes co-design: diverse sandboxes paired with real-device runtimes, an action space that mixes GUI and command-line operations, data flywheels where agents build their own training tasks, and a training mix that includes online RL at scale. Online RL's role is to close the gap between benchmark and real world, but it only works if the environments, data and action space are built to feed it.
There is good reason to think RL can carry that load. Can reinforcement learning scale beyond single-turn language tasks? shows RL working in long, stateful tasks where the reward arrives only at the end. Modified DAPO training doubled SWE-bench Verified performance from 20% to 39% on Qwen2.5-72B, matching larger models. GUI work has the same shape: many steps, a changing screen, and success known only at the finish. This is a neighboring domain, so it shows the tool works in this kind of setting. It is not a GUI result.
The catch is that online RL needs a live environment to practice in, and that is expensive. Can LLMs replace search engines during agent training? shows one workaround for search agents: LLMs playing the search engine, with 14B simulators matching or beating real engines, so training runs without API bills. Nothing in the collection applies this to GUIs. But it names the cost pressure that the GUI note's emphasis on diverse sandboxes seems to be responding to.
Online RL is also the expensive lever. Do self-improving agents really split into two distinct loops? splits agent improvement into a slow loop that updates model weights, where RL lives, and a fast loop that updates prompts, memory and tools. Recent progress concentrates in the fast loop because it is cheaper and reversible. Can agents learn continuously from experience without updating weights? shows the RL framing can run on memory alone, reaching 87.88% on GAIA validation with no weight changes. Can execution harnesses lift model performance without retuning weights? lifts frozen models by optimizing the system around them. Can API-first agents outperform UI-based agent interaction? cuts task time 65-70% by skipping UI clicking where an API exists. Scaling GUI agents likely blends these cheaper levers with RL. The collection has no head-to-head on one GUI benchmark, so how much of the gain comes from RL versus scaffolding and shortcuts is still open here.
Sources 7 notes
Building effective GUI agents requires systems-level co-design across multiple components: diverse sandboxes paired with real-device runtimes, unified action spaces combining GUI and CLI operations, data flywheels using agents to construct tasks, and combined training approaches including online RL at scale.
Modified DAPO training doubled SWE-bench Verified performance from 20% to 39% on Qwen2.5-72B, matching larger models. This demonstrates RL works in stateful multi-step environments with delayed rewards and complex feedback, beyond theoretical single-turn MDPs.
ZeroSearch and SSRL demonstrate that LLMs can generate relevant documents and search results from internal knowledge, with 14B simulators matching or exceeding real search engines. Curriculum degradation and test-time scaling optimize this approach for training without API costs.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Show all 7 sources
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
The AXIS framework shows that prioritizing API calls over sequential UI interactions cuts task completion time by 65–70% while maintaining 97–98% accuracy and reducing cognitive workload by 38–53%. A self-exploration mechanism automatically discovers and constructs APIs from existing applications, solving the bootstrapping problem.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
- LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Self-Improvements in Modern Agentic Systems: A Survey
- Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning
- Useful Memories Become Faulty When Continuously Updated by LLMs
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- ZeroSearch: Incentivize the Search Capability of LLMs without Searching