TOPIC
Tool Use and Computer-Use Agents
A subject the collection covers, read through 13 synthesis notes.
View as
Can small models match large models on function calling?
Explores whether small language models fine-tuned with the right training method can achieve comparable performance to large models on structured reasoning tasks requiring precise function calls, and what training approach makes this possible.
Can small language models handle most agent tasks? Can breaking function calling into subtasks improve model generalization? Where do traditional function calling systems actually break down? Does teacher-refined data always improve student model performance? Why do alignment methods work if they model human irrationality?
Small language models suffice for most agentic subtasks cost-effectively Function calling decomposes into seven subtasks that multi-task training addresses better Function calling fails at three independent points, not one Student models must selectively incorporate teacher refinements based on compatibility Alignment methods work by modeling human cognitive biases like loss aversion
Can communication problems and optimization problems be addressed with the same alignment approaches? Why does DPO outperform SFT specifically for function calling tasks? Can granular sub-task training for function calling improve both open and proprietary models? When does the right constraint beat additional model capacity? Can input augmentation and rephrasing compensate for smaller model limitations? How do larger models maintain more parallel tasks than smaller models? Can likelihood choice matter more than architectural depth for CF? Do models learn different sophistry strategies for QA versus code generation? Why do method-level improvements avoid the generation-verification gap that parameter-level improvements face? Does correct model behavior guarantee internal alignment of learned objectives? Can language models learn to form ad-hoc conventions through training? How much of the combinatorial task space must training data cover?See all 113 inquiring lines on this note →
Can structured reasoning replace code execution for RL rewards?
Can semi-formal templates enable execution-free code verification reliable enough to train RL agents without running code? This matters because execution is expensive and slow in agent training loops.
Can structured templates make code reasoning more reliable than free-form thinking? Can structured templates replace formal verification for code reasoning? Can step-wise expert rewards help small models learn hard reasoning? Can code serve as the operational substrate for agent reasoning? Can reasoning improvement work without answer verification?
Structured reasoning templates force completeness and prevent unstructured reasoning failures Structured templates bridge formal verification and unstructured reasoning effectively Step-wise expert-similarity rewards enable small models to learn hard reasoning Code operates as executable inspectable stateful substrate for agent systems Reference answer likelihood replaces verification in general-domain reasoning RL
What makes some tasks bounded enough for reliable RL? How does the execution layer constrain agent performance in tool use? Can external verifiers replace reasoning trace quality in solution guarantees? Can structured output formats reduce instruction following degradation? What infrastructure decouples generation from training in asynchronous agent loops? What makes software engineering environments better suited for RL than other interactive domains? Can exchange value persist without use value being verified first? What execution-layer design prevents agents from passively reacting to environments? How should humans specify deterministic abstractions of RL problems? How should harness infrastructure validate code that agents generate themselves? When should agent-created code be promoted into permanent harness infrastructure? How should we measure context efficiency and verification cost in agents?See all 38 inquiring lines on this note →
How can GUI agents adapt when software constantly changes?
Can desktop automation agents stay current by combining real-time web documentation with learned task patterns and concrete execution memories? This explores how to avoid training obsolescence in open-world software environments.
Can structured interfaces help language models control GUIs better? Can agents learn preferences by watching rather than asking? Can reasoning systems maintain memory across retrieval cycles? Does state-indexed memory outperform high-level workflow memory for web agents? Why do planning and grounding pull against each other in agents?
Structured interfaces with vision plus accessibility trees beat raw screenshots Entity-centric memory graphs enable preference learning from continuous multimodal observation Stateful memory workspaces enable narrative reasoning across retrieval cycles State-dependent procedural memory outperforms workflow-level abstractions for web agents Foundation GUI agents need an intermediate interface separating planning from grounding
How should GUI agents remember patterns across different software environments? Can episodic memory of UI traces improve open-world agent adaptation? Can this approach handle continuously changing product inventories in production? What makes idle window detection valuable for continuous agent improvement?
Can breaking function calling into subtasks improve model generalization?
Does training on seven granular function-calling subtasks instead of one umbrella objective close the gap between open-source and proprietary models? This explores whether decomposition surfaces hidden failure modes that unified training misses.
Where do traditional function calling systems actually break down? Can small models match large models on function calling? Does training order reshape how models handle different task types? Does separating planning from execution improve reasoning accuracy? Can small language models handle most agent tasks?
Function calling fails at three independent points, not one DPO training on teacher-generated preference pairs outperforms SFT for small models Training order mechanically shapes entropy dynamics across domain types Separating decomposer from solver improves accuracy and generalizability Small language models suffice for most agentic subtasks cost-effectively
Why does DPO outperform SFT specifically for function calling tasks? What role does rigid output format play in function calling failure modes? Can granular sub-task training for function calling improve both open and proprietary models? What makes the frame problem distinct from feature-level shortcuts? How much of the combinatorial task space must training data cover? Why does full multi-task fine-tuning perform worse than sequential training? What task structures benefit most from geometric parameter merging? How does preference-based training compare to supervised fine-tuning for function calling? Why do single function-calling benchmarks mask model weakness in specific areas? Do different function-calling subtasks have different entropy profiles during training? What three independent failure points bottleneck traditional function calling systems? How do ensemble methods apply within a single model?See all 27 inquiring lines on this note →
Can structured interfaces help language models control GUIs better?
Explores whether separating visual understanding from element grounding through an intermediate interface layer improves how language models interact with graphical interfaces. Matters because current end-to-end approaches ask models to do too much at once.
Why do planning and grounding pull against each other in agents? Why do vision-only GUI agents struggle with screen interpretation? How can GUI agents adapt when software constantly changes? Do text-based GUI agents actually work in the real world? Can API-first agents outperform UI-based agent interaction?
Foundation GUI agents need an intermediate interface separating planning from grounding Structured screen parsing unblocks vision-language models for GUI tasks Stratified memory architecture lets GUI agents adapt to evolving software Visual perception requires UI-specialized models, not general multimodal ones API-first agents reduce task completion time by 65 to 70 percent
Can better AI interfaces eliminate the attention cost of prompt composition and evaluation? What scaffolding tools help users specify implicit contextual boundaries to models? Can parsing screens into structured elements before acting improve vision models? What role does visual perception play alongside accessibility tree information? What makes the frame problem distinct from feature-level shortcuts? How do standardized artifacts improve coordination between multiple tools? How does spatial density in web UIs break workflow-level memory? Why does explicit screen parsing outperform pure vision in GUI agents? What other semantic relations benefit from explicit surface markers in text? Can designers hide AI context complexity behind a stable user interface? How should designers make invisible AI state legible to users? What does an intermediate interface between planning and grounding actually look like?See all 37 inquiring lines on this note →
Can unlabeled UI video teach models what users intend?
Can temporal masking on screen recordings learn task-aware representations without paired text labels? This matters because labeled UI video is scarce and expensive, so self-supervised learning could unlock scaling.
Why do vision-only GUI agents struggle with screen interpretation? Do text-based GUI agents actually work in the real world? How can GUI agents adapt when software constantly changes? Can models reason without generating visible thinking steps? Can careful curation replace massive alignment datasets?
Structured screen parsing unblocks vision-language models for GUI tasks Visual perception requires UI-specialized models, not general multimodal ones Stratified memory architecture lets GUI agents adapt to evolving software Models reason effectively in latent space without verbalized tokens Alignment quality depends far more on data curation than quantity
Can parsing screens into structured elements before acting improve vision models? How much does autonomous action without prompting affect user perception? Why does explicit screen parsing outperform pure vision in GUI agents? Can agents learn user intent from unlabeled video without text labels? What makes a self-supervised pruning metric work without labels at scale? Why does pure-vision underperform when parsing semantics and action prediction mix? What role does prediction error play in human event segmentation? Can temporal ranking improve retrieval without modifying the underlying video model? Why do static screenshot models fail to capture multi-step UI task intent? What temporal signals in screen recordings matter most for task understanding? What makes accessibility trees insufficient compared to visual GUI understanding? Can self-supervised process models replace human annotations at scale?See all 23 inquiring lines on this note →
Can models decide better than retrievers which tools to use?
Traditional retrieval picks tools upfront based on initial queries, but do models themselves make better decisions about tool needs as they reason? This explores whether authority over tool selection should move from external systems to the LLM.
Function calling fails at three independent points, not one Agent attention economy will replace user attention competition Models can be trained to proactively identify missing information and request clarification Decoupling reasoning from observations eliminates redundancy and enables parallelism Agent capability alone cannot ensure successful real-world deployment
Should LLMs query users back when presented with under-specified scenarios? What makes proactive tool retrieval better than single-round semantic matching? How should moderator LLMs decide which speakers to query per topic? Should retrieval be triggered always or only for difficult questions? Do different domains require different types of model investment? How can smaller models help select useful data for larger models? Could eliminating retrieval entirely work better than shifting the burden? How does semantic mismatch between user language and API documentation degrade tool retrieval? When should a system decide to retrieve versus reason alone? How does semantic clustering help decide which model handles each query? How do agents discover and select which tools to invoke? Can models retrieve the right tool without relying on vector similarity?See all 23 inquiring lines on this note →
Can structured templates make code reasoning more reliable than free-form thinking?
Unstructured chain-of-thought reasoning lets models skip cases and make unsupported claims. This explores whether semi-formal templates requiring explicit premises, evidence traces, and alternative checks can prevent these failure modes.
Can structured templates replace formal verification for code reasoning? Can structured reasoning replace code execution for RL rewards? Can structured argument prompts make LLM reasoning more rigorous? Does chain-of-thought reasoning reveal genuine inference or pattern matching? Where do reasoning agents actually fail during long traces?
Structured templates bridge formal verification and unstructured reasoning effectively Structured reasoning reaches 93% accuracy for execution-free code verification Structured critical questions force LLMs to check warrants and improve reasoning Chain-of-thought mimics reasoning form through learned schemata, not genuine inference Process verification catches errors that final-answer scoring misses entirely
Can contextual design decisions resist formalization into evaluation rubrics? Can reasoning in free text then formatting separately recover performance? Can static reasoning patterns work better than dynamic branch selection? What makes structured memory schemas more stable than freeform text summaries? How do execution traces represent state and dynamics in codebase modeling? Which code verification tasks still require execution instead of reasoning? Why do semi-formal templates improve verification accuracy over unstructured reasoning? What makes structured stochasticity more effective than unstructured randomness in reasoning? Why does unstructured chain-of-thought permit assumption-based errors that templates prevent? Can completeness scaffolding substitute for actual code execution in reasoning? How can structured reasoning templates serve as rewards for code agent training? What makes natural language reasoning more practical than formal languages for multi-framework codebases?See all 16 inquiring lines on this note →
Does state-indexed memory outperform high-level workflow memory for web agents?
Should procedural memory for web agents be organized around specific environment states and actions, or abstracted into higher-level workflows? This matters because web automation demands precise, context-sensitive recall that workflows might lose.
Can agents learn reusable sub-task routines from past experience? Can frozen language models continually improve through memory structure alone? How can GUI agents adapt when software constantly changes? Can agents learn preferences by watching rather than asking? Why do LLM agents ignore condensed experience summaries?
Agents gain 24-51 percent by extracting and compounding reusable sub-task workflows Causal abstractions in memory let frozen agents continually improve and transfer Stratified memory architecture lets GUI agents adapt to evolving software Entity-centric memory graphs enable preference learning from continuous multimodal observation LLM agents depend on raw experience but disregard condensed experience
How does spatial density in web UIs break workflow-level memory? How do the six memory components combine across explicit and implicit paths? Why does GUI agent memory need different abstraction levels? How should headers index procedural intent differently from keyword chunking? Can state-indexed memory retrieval breadth predict gains in web agent robustness? What makes a memory reachable in the right context? How does procedural memory granularity affect web agent performance? How does workflow abstraction compare to state-indexed procedural memory for web agents? What is the right granularity level for agent memory to enable both reuse and composition? Does workflow-level memory or state-action memory better capture reusable agent knowledge? How do strategy-level abstractions differ from storing raw task workflows? How do memory-resident safeguards get surfaced at the exact decision point where they matter?See all 21 inquiring lines on this note →
Can structured templates replace formal verification for code reasoning?
Formal verification is rigorous but impractical at repository scale. Can natural-language templates with enforced structure provide the same reliability guarantees without the formalization cost? This explores the middle ground between unstructured reasoning and full formalism.
Can structured templates make code reasoning more reliable than free-form thinking? Can structured reasoning replace code execution for RL rewards? Can symbolic solvers fix how LLMs reason about logic? Should LLMs handle abstraction only in optimization? Can code serve as the operational substrate for agent reasoning?
Structured reasoning templates force completeness and prevent unstructured reasoning failures Structured reasoning reaches 93% accuracy for execution-free code verification Symbolic solver integration enables reliable logical reasoning through structured feedback LLM optimization works best when restricted to abstraction only Code operates as executable inspectable stateful substrate for agent systems
Should validation responsibility move away from the primary user? What are the five structure types and which tasks does each one suit best? Does the verification gap widen exactly where judgment replaces checkability? Which code verification tasks still require execution instead of reasoning? Why do semi-formal templates improve verification accuracy over unstructured reasoning? Can partial formal verification work without full formalization of language semantics? What makes natural language reasoning more practical than formal languages for multi-framework codebases? Can completeness scaffolding work for domains beyond code verification? What makes code inspectable feedback more reliable than natural language verification? What makes structured informal reasoning preferable to full formalization?
Does agent interaction time scale separately from reasoning depth?
Can agents improve by taking more environment steps rather than thinking harder per step? This matters because partially observable tasks like web navigation may need exploration and backtracking that deeper reasoning alone cannot provide.
How should we balance parallel versus sequential compute at test time? Does search budget scale like reasoning tokens for answer quality? Does gradually tightening token budgets beat fixed budget training? Why does parallel reasoning outperform single chain thinking? How should we spend compute at inference time?
Parallel and sequential scaling represent a fundamental trade-off in test-time compute Search budget exhibits test-time scaling identical to reasoning tokens Curriculum budgets starting generous then tightening outperform fixed budget training Parallel reasoning paths outperform extended single chains consistently Inference compute allocation must adapt to prompt difficulty
Why do planning and grounding have opposing optimization requirements in agents? When should you optimize agent behavior versus tool performance separately? What accounts for performance drops in multi-turn agent interactions? How should agents separate planning from perception grounding? What does an intermediate interface between planning and grounding actually look like? Does the planning-grounding factoring principle apply to other agent tasks? How should the surrounding agent system be designed to ground actions in reality? Can extended deliberation in agents become counterproductive like human overthinking? How do planning and grounding have opposing optimization requirements in agents? Can curriculum approaches teach agents when to stop exploring? Why does partial observability require interaction instead of better reasoning? Should agent capability be optimized separately from general capability?See all 17 inquiring lines on this note →
Will agents compete for attention just like users do?
As autonomous agents take over user tasks, will the Web's economic competition shift from human clicks to agent invocations? This explores whether existing ad-market mechanisms could scale to agent decision-making.
Models should request tools proactively instead of passive retrieval selecting them API-first agents reduce task completion time by 65 to 70 percent Agent capability alone cannot ensure successful real-world deployment Generated task-specific UIs outperform chat in over 70 percent of cases Small language models suffice for most agentic subtasks cost-effectively
Can sorting algorithms create symmetric competition between human and AI content? What happens when tools compete for agent invocation rather than human clicks? What makes a service visible to autonomous agent systems? Can advertising mechanisms designed for humans work on agents? What ecosystem conditions make agent attention markets viable? How will the agent economy reshape compute infrastructure design? What economic incentives make advertisement embedding attacks persistently viable?
Where do traditional function calling systems actually break down?
Function calling seems simple but fails in ways that aren't obvious. This explores three independent failure points—retrieval, context bloat, and output rigidity—that together explain why even the best models struggle.
Can models decide better than retrievers which tools to use? Can breaking function calling into subtasks improve model generalization? Can small models match large models on function calling? Can reasoning and tool execution be truly decoupled? Why does random tool sampling produce unrealistic synthetic training data?
Models should request tools proactively instead of passive retrieval selecting them Function calling decomposes into seven subtasks that multi-task training addresses better DPO training on teacher-generated preference pairs outperforms SFT for small models Decoupling reasoning from observations eliminates redundancy and enables parallelism Graph-based sampling and planned dialogue restore realism in synthetic tool-calling data
Why does DPO outperform SFT specifically for function calling tasks? What role does rigid output format play in function calling failure modes? Why do single function-calling benchmarks mask model weakness in specific areas? What three independent failure points bottleneck traditional function calling systems?