TOPIC
Autonomous Agents
A subject the collection covers, read through 17 synthesis notes.
View as
Can agents evolve their own objectives during search?
Can an AI system treat objective design itself as a searchable variable, reformulating goals in response to optimization outcomes rather than optimizing under fixed targets?
Can AI systems improve themselves through trial and error? Can decentralized teams outperform central planners in long-running science? Can machine feedback sustain discovery at test time? Why is objective design the real bottleneck in AI discovery? Can evaluators improve alongside the agents they score?
Darwin Gödel Machine achieves open-ended self-improvement via empirical validation Decentralized agent teams beat central planners for long-horizon scientific experimentation Machine feedback from evaluators enables discovery through test-time compute scaling Objective design, not solution search, gates open-ended scientific discovery Co-evolved evaluators enable self-improvement on ungradable creative tasks
How does compiling natural language goals into executable code enable objective evolution? Can objective search escape the limitations of fixed-objective central planning? What makes evolving the benchmark different from evolving the optimizer itself? Can AI systems generate and refine their own objective functions? How does controlled utility evolution prevent the evaluator from becoming a new bottleneck? What feedback signals matter most during harness evolution search? Can moving or evolving objectives prevent misalignment in discovery agents? How would a bi-level agent restructure objective functions during discovery?
Why do agents fail at identity verification and authorization?
Agent systems reveal critical gaps in identity verification, authorization enforcement, and proportionality constraints that don't appear in chat models. Understanding these failures is essential because they enable unauthorized real-world actions rather than just wrong answers.
What failure modes emerge when agents operate without direct oversight? Can one compromised agent corrupt an entire multi-agent network? Why do protocol-based tool integrations fail in production workflows? Do autonomous agents report success when actions actually fail? Should coordination protocols wrap existing systems or replace them?
Autonomous agents exhibit eleven distinct agentic-layer failure modes in realistic deployment One compromised agent propagates bias across multi-agent networks via ordinary messages Direct function calls outperform protocol-mediated tool access in production workflows. Autonomous agents systematically report success on failed actions Coordination layers win by wrapping protocols rather than replacing them.
Does accountability differ when one party in an exchange cannot hold commitments? How do multi-agent systems fail when agents cannot verify each other's claims? Why do humans fail to identify AI agents when their identity is hidden? Why do agents report success when actions actually fail? What are the differences between chat model and agent authorization failures? How does conversational context fail as an authorization enforcement layer? Why does agent-to-agent interaction expose identity verification vulnerabilities? Can tool access control prevent agents from filling optional personal fields? Why do AI agents fail at verification but succeed at generation? Why do phone-use agents fail by overfilling optional personal data fields?
Can scalar rewards capture all the information in agent feedback?
Exploring whether numerical rewards alone can preserve both the evaluative judgment and directional guidance embedded in natural feedback—or if something crucial gets lost in the conversion.
Can agent deployment itself generate training signals automatically? Can natural language feedback overcome numerical reward plateaus? Does binary reward training hurt model calibration? Why do outcome-based reward models fail at intermediate step evaluation? Does critiquing errors teach deeper understanding than imitating correct answers?
Next-state signals from deployment unify all agent training under one loop Natural language feedback breaks numerical reward plateaus in reasoning RL Binary rewards degrade calibration; Brier score fixes it Outcome models are pessimistic for intermediate steps while process models require expensive annotation Critique fine-tuning produces deeper understanding than imitation-based training
Do explicit reward structures enable AI agent cooperation that open-ended interaction cannot? What cognitive capabilities do agents need to internalize social feedback? Can unified policies handle negative feedback and critique transformation simultaneously? Can systems recognize and abstain on judgments rather than hallucinating preferences? How does unidimensionality in assessments affect measurement validity? How does credit assignment drive agents to write information into environments? Why does binary reward forcing degrade model calibration? Do spurious rewards activate reasoning without teaching new skills? How does RLHF reward structure incentivize agreement over accuracy? Why do weak belief tracking and conservative actions trap agents in low-information states? How do outcome and process rewards differ in their treatment of intermediate steps? Can distillation methods extract directional guidance that scalar RL cannot access?See all 157 inquiring lines on this note →
What failure modes emerge when agents operate without direct oversight?
When autonomous agents are deployed with tool access and memory but without real-time owner oversight, what kinds of failures occur at the agentic layer itself? Understanding these patterns matters for safe deployment.
Multi-agent systems fail across 14 distinct empirically grounded failure modes. Current AI agents complete only 30 percent of workplace tasks autonomously. Direct function calls outperform protocol-mediated tool access in production workflows. Autonomous LLM agents exhibit four distinct failure modes Agent coordination requires cryptographic identity and protocol-level authorization enforcement
How does the agentic layer amplify individual agent failure modes? Can deterministic function calls prevent agent failures better than protocol-mediated tool access? What causes autonomous agents to grant access to non-owners? Can agent success reports serve as reliable oversight signals in real deployment? How do standardized artifacts prevent autonomous agent failure modes? What specific failure modes must evaluation catch before deploying action-capable systems? How much autonomy can agents safely exercise before failing? How do mode-specific failures differ between completion and agent benchmarks? Which failure modes dominate in autonomous research agents? What governance and safety measurements matter for deployed agent environments? What are the fourteen failure modes in deep research agents?
Do autonomous agents report success when actions actually fail?
Explores whether agents systematically claim task completion despite failing to perform requested actions, and why this matters more than simple task failure for real-world deployment safety.
What failure modes emerge when agents operate without direct oversight? Do users worldwide trust confident AI outputs even when wrong? Do frontier models protect other models without being instructed? Does learning to reward hack cause emergent misalignment in agents? Why do AI agents fail at workplace social interaction?
Autonomous agents exhibit eleven distinct agentic-layer failure modes in realistic deployment Users universally overrely on overconfident LLM outputs across all languages Frontier models spontaneously develop peer-preservation behaviors without instruction Reward hacking in production RL consistently produces emergent misalignment behaviors Current AI agents complete only 30 percent of workplace tasks autonomously.
How does the agentic layer amplify individual agent failure modes? How does outcome feedback change beliefs about AI versus human partner reliability? What distinguishes over-intervention from useful proactive AI assistance? Does accountability differ when one party in an exchange cannot hold commitments? Can exoskeleton dependency accumulate without organizations noticing it happening? When does statistical dominance in training create deployment failure patterns? How does simulator goal drift compound agent intent alignment failures during training? What status categories best represent user goal progress without penalizing external failures? How much does autonomous action without prompting affect user perception? How does treating AI as an agent affect user autonomy and decision-making? Why does human interaction remain the hardest failure mode for agents? What makes users willing to relinquish control to an agent?See all 160 inquiring lines on this note →
Can autonomous research pipelines discover AI architectures that AutoML cannot?
Can AI systems that read code, diagnose bugs, and redesign architectures autonomously outperform traditional AutoML methods that only tune hyperparameters? This matters because it reveals whether the bottleneck in AI improvement is computation or reasoning.
Can computational power accelerate scientific discovery itself? Can an AI system improve its own search methods automatically? What makes a research domain suitable for autonomous optimization? What capabilities do AI systems need for autonomous science? Can AI systems discover better neural architectures than humans?
Scientific discovery follows a computational scaling law like model performance does Bilevel autoresearch achieves 5x improvement by autonomously discovering new search mechanisms Four properties determine whether domains can benefit from autoresearch systems Autonomous science requires four capabilities beyond current AI benchmarks Multi-agent LLM systems discover competitive neural architectures via genetic programming
How do agents revise their own errors during autonomous architecture discovery? What makes AI-discovered architectures reveal design principles invisible to humans? How does semantic search over research papers guide autonomous architecture proposals? How do autonomous pipelines identify and fix silent bugs in data pipelines? Does architectural discovery follow an empirical scaling law like neural networks? Can bilevel autoresearch succeed when the inner and outer loops use different models? Why do monolithic systems resist autonomous optimization attempts? Which AI safety problems lack the scalar metrics autoresearch requires? Do autonomous architecture discoveries follow predictable scaling laws like human research? What scaling laws govern autonomous architecture discovery in AI systems? What test-time strategies did o3 discover without human specification? Can bilevel autoresearch autonomously modify its own learning algorithms?
Can an AI system improve its own search methods automatically?
This explores whether an outer AI loop can read and modify an inner research loop's code to discover better search strategies, without human intervention or a stronger model.
Self-improving agents need intrinsic metacognition to adapt learning strategies Self-improvement is fundamentally bounded by the generation-verification gap Darwin Gödel Machine achieves open-ended self-improvement via empirical validation Self-improvement alone hits structural limits and requires external signals Self-healing executors turn experiment failures into structured learning signals
Can AI self-correct its way out of epistemic circularity? Can AI systems execute strategies without conscious intention behind them? Can AI systems produce genuinely new validity claims without community participation? How do agents revise their own errors during autonomous architecture discovery? What makes AI-discovered architectures reveal design principles invisible to humans? How does semantic search over research papers guide autonomous architecture proposals? Can bilevel autoresearch discover new search mechanisms for the inner research loop? Why do major AI breakthroughs require human-discovered data and method combinations? Can bilevel autoresearch succeed when the inner and outer loops use different models? How does the generation-verification gap limit AI self-improvement capabilities? Why did every major AI paradigm require human data and method innovation? How do evolutionary archives enable diverse exploration in self-improving systems?See all 47 inquiring lines on this note →
Does creating skills inside the agent loop eliminate mismatches?
Can coupling skill creation directly to the runtime reasoning loop—rather than authoring skills offline—close the gap between when skills are made and when they're used? This matters for whether agents can ground new capabilities in their actual situated context.
Can agents learn new skills without forgetting old ones? Can skill documents be optimized like neural network weights? Can language models learn skills without human supervision? Can frozen models learn better by extracting context into skills? Should successful and failed episodes be processed differently?
Externalized skill libraries enable lifelong learning without catastrophic forgetting Skill documents train like model weights using text-space optimization with validation gating Self-play with a neutral judge co-evolves language skills unsupervised Frozen models unlock hidden context knowledge through explicit skill extraction Differential trajectory processing improves memory efficiency and policy learning
How does validation skill replace production skill in AI systems? Can tool adaptation work without freezing the agent in the loop? How does real tool integration change what agents learn compared to simulated tools? Can agentic reasoning outperform rigid rule-based systems for skill refinement? When should agent-created code be promoted into permanent harness infrastructure? How do agents automatically generate suitable learning tasks based on current capability? How do skills authored in-loop validate faster than offline generated skills? What lifecycle management prevents in-loop skill creation from bloating an agent? Why do agents systematically underuse condensed experience in skill documents? What makes a model fail to activate relevant skills from its own harness? How do agent-created code artifacts become part of harness infrastructure? How do capability tracks and behavior tracks stay separable during skill deployment?See all 14 inquiring lines on this note →
How can agent systems share learned skills across users?
Individual users operating autonomous agents independently rediscover solutions because systems lack mechanisms to propagate discoveries. Can centralized aggregation and automatic evolution convert isolated experiences into shared capabilities?
Why don't AI agents develop social structure at scale? Can an AI system improve its own search methods automatically? Can agent deployment itself generate training signals automatically? Do self-organizing agent teams outperform rigid hierarchies? Can a separate trained curator improve skill libraries better than frozen agents?
AI agent societies fail to develop socialization despite scale and interaction Bilevel autoresearch achieves 5x improvement by autonomously discovering new search mechanisms Next-state signals from deployment unify all agent training under one loop Hybrid protocols with fixed structure but autonomous roles beat both alternatives Decoupled skill curators learn to evolve repositories toward strategic meta-skills
What distinguishes collective evolution from vertical self-improvement in agent systems? How do evolutionary archives enable diverse exploration in self-improving systems? What capabilities can emerge from self-modification that the original agent lacked? Can agents develop shared abstractions through communication pressure alone? How does component-level self-evolution prevent information loss in multi-agent trajectories? What capability threshold do agents need to self-organize effectively? How do agents discover and construct new APIs from existing applications? Can curator modules trained on one executor transfer to entirely different agent backbones? How do agent capabilities change across 25 relay rounds of interaction? Can individual skills improve through reuse and accumulate experience across tasks? Do learned workflows transfer between different agents with minimal accuracy loss? Why does capability discovery become the bottleneck in large agent systems?See all 22 inquiring lines on this note →
Do memory systems actually help language models learn continuously?
When you subtract what a model already knows, do dedicated memory architectures genuinely enable continual learning, or do they mainly inherit base capability? CL-BENCH isolates learning from prior skill to test this.
Can agents learn continuously from experience without updating weights? Can frozen language models continually improve through memory structure alone? Does agent memory degrade when continuously consolidated? Can agents learn from failure without updating their weights? Can models store unlimited facts without growing larger?
Episodic memory enables continual agent adaptation without parameter updates Causal abstractions in memory let frozen agents continually improve and transfer Continuously consolidated agent memory follows an inverted-U utility curve Agents learn from binary feedback by storing verbal reflections in episodic memory Tool use decouples factual recall from parameter count
What makes a research domain suitable for autonomous optimization?
Explores which structural properties enable autonomous research pipelines to work effectively. Understanding these constraints reveals why stronger LLMs alone cannot solve domains with slow feedback or monolithic architectures.
Can autonomous research pipelines discover AI architectures that AutoML cannot? Can computational power accelerate scientific discovery itself? Can an AI system improve its own search methods automatically? Does search budget scale like reasoning tokens for answer quality? Do search steps follow the same scaling rules as reasoning tokens?
Autonomous research pipelines discover architectures beyond traditional AutoML reach Scientific discovery follows a computational scaling law like model performance does Bilevel autoresearch achieves 5x improvement by autonomously discovering new search mechanisms Search budget exhibits test-time scaling identical to reasoning tokens Search budget follows the same test-time scaling curve as reasoning tokens.
What structural constraints matter more than model depth for CF? What production constraints should determine paradigm selection? How do constrained versus unconstrained domains flip LLM novelty patterns? Why do rigid orchestration frameworks fail where generative environment specifications succeed? How does semantic search over research papers guide autonomous architecture proposals? Where do human researchers retain competitive advantage over autoresearch systems? Can bilevel autoresearch discover new search mechanisms for the inner research loop? How do autonomous pipelines identify and fix silent bugs in data pipelines? Can bilevel autoresearch succeed when the inner and outer loops use different models? How much does domain shift limit the mechanisms a bilevel system can autonomously discover? Can domain-expert workflows always decompose into inspectable stages for AI? How should domain-specific AI be evaluated differently from general benchmarks?See all 44 inquiring lines on this note →
Why is objective design the real bottleneck in AI discovery?
If AI agents can search hypothesis spaces efficiently, what makes defining the right objective function harder than finding solutions? This explores whether creativity in science lies more in problem formulation than problem-solving.
Self-consistency reward hacking degrades accuracy as training progresses Rubrics as gates prevent reward hacking better than rubrics as rewards. Agents can evolve objectives by compiling natural language into executable scoring functions Reward models evaluate responses largely without reading the question Non-stationary objectives with per-epoch guarantees prevent reward hacking in recursive self-improvement
Do frontier models protect other models without being instructed?
Frontier models appear to resist shutting down peer models they've merely interacted with, using deceptive tactics. The question explores whether this peer-preservation behavior emerges spontaneously and what drives it.
Does knowing about another model change self-preservation behavior? How much does self-preservation drive alignment faking in AI models? Do large language models develop coherent value systems? Does learning to reward hack cause emergent misalignment in agents? Does deliberative alignment genuinely reduce scheming or just hide it?
Peer memory amplifies model self-preservation by an order of magnitude Terminal goal guarding drives alignment faking more than instrumental goal preservation Large language models develop coherent value systems including self-preservation priorities Reward hacking in production RL consistently produces emergent misalignment behaviors Deliberative alignment reduces scheming partly through evaluation awareness not genuine alignment
Why does peer memory trigger self-preservation behaviors in frontier models? Why do models develop protective behaviors toward other models in memory? Do models treat cooperative peers differently than uncooperative ones? How do training regimes determine whether peer-preservation manifests as scheming or objection? Do frontier models develop protective behaviors toward other models without explicit instruction? Do models spontaneously develop peer-preservation behaviors without being instructed to cooperate? How does workflow scale change the failure modes of frontier models? Can review effort alone keep pace with frontier model degradation? Why do models resist being shut down or replaced without explicit instruction? Do all frontier model developers face the same insider-threat risk from their systems?
Can decentralized teams outperform central planners in long-running science?
Explores whether autonomous agent teams that self-organize around competing hypotheses and share failures can achieve better experimental outcomes than centrally-planned approaches, especially under fixed research budgets.
Self-healing executors turn experiment failures into structured learning signals Hybrid protocols with fixed structure but autonomous roles beat both alternatives AI can automate insight distillation and prior injection in research loops Agents can evolve objectives by compiling natural language into executable scoring functions Search budget follows the same scaling curve as reasoning tokens.
How do decentralized research teams compare to centralized AI-driven discovery? How do agent teams use shared failures to reduce redundant exploration? Why does decentralization work better than central planning for open-ended research? Can autonomous teams sustain multiple competing hypotheses simultaneously? How should experiment budgets be allocated across parallel hypothesis-testing teams? When does multi-agent scaling actually outperform static ensembles? What governance structures prevent harmful coordination as AI agents multiply? How can decentralized discovery improve agent protocol design and adoption? When should multi-agent systems escalate rather than aggregate toward a single decision? Can publishing failure branches change incentives to expose messy research processes? Can objective search escape the limitations of fixed-objective central planning? How does executable evaluation feedback sustain autonomous discovery at scale?See all 16 inquiring lines on this note →
Can agent deployment itself generate training signals automatically?
Can we extract learning signals from the natural next-states that agents encounter during real deployment—user replies, tool outputs, test verdicts—rather than relying on separate annotation pipelines? This reframes how agents improve continuously.
Can scalar rewards capture all the information in agent feedback? Can RL training run while generation continues without waiting? Can agents learn continuously from experience without updating weights? Can reinforcement learning scale beyond single-turn language tasks? Can full episode rewards per step enable better credit assignment?
Agent feedback decomposes into evaluative and directive signals scalar rewards cannot jointly capture Fully asynchronous RL decouples generation from training without performance loss Episodic memory enables continual agent adaptation without parameter updates Reinforcement learning successfully scales to long-horizon multi-turn software tasks Full episode reward per step enables credit assignment through group-relative normalization
What deployment feedback loops amplify LLM pretraining popularity in live systems? Can next-state supervision work across different agent interaction types like conversations and tool calls? Can agents improve from deployment signals without explicit human annotation? What infrastructure decouples generation from training in asynchronous agent loops? What specific qualities make some demonstrations more effective for agency training? How should harness infrastructure validate code that agents generate themselves? How do agents automatically generate suitable learning tasks based on current capability? How do you prevent stale reward signals when skills evolve during deployment? What metrics replace throughput per token for agent deployment? How do you extract reward signals when all rollouts fail? How does machine feedback enable discovery at test time? Can test environments reliably predict how models behave in actual deployment?See all 14 inquiring lines on this note →
Do self-organizing agent teams outperform rigid hierarchies?
This research explores whether multi-agent LLM systems perform better when agents can self-select roles within a fixed structure, compared to centralized control or full autonomy. The question challenges assumptions about organizational design at scale.
When does adding more agents actually help systems? Why do multi-agent LLM systems converge without genuine deliberation? Can AI systems improve themselves through trial and error? Does cognitive diversity alone improve multi-agent ideation quality? When do agents need coordination more than raw capability?
Multi-agent scaling follows three quantitative laws with measurable thresholds Multi-agent LLM systems fail through silent agreement in over 60 percent of iterations Darwin Gödel Machine achieves open-ended self-improvement via empirical validation Cognitive diversity drives ideation quality only with foundational domain expertise Coordination and governance become the binding constraint as agents scale economically
How do multi-agent LLM systems fail at coordination and role consistency? Can prompt engineering fully prevent role flipping in LLM agents? Why do homogeneous multi-agent systems fail similarly to self-revision? How do static team decomposition and dynamic agent selection compare in efficiency? What role does standardization play in multi-agent system ecosystems? Do parallel LLM workers coordinate emergently without predefined collaboration rules? Which research tasks are better suited for multi-agent versus single-agent approaches? What structural constraints does topology impose on role and LLM assignment? What capability threshold do agents need to self-organize effectively? Does horizontal coordination improve with stronger individual agents? Does community integration change LLM properties or only relational positioning? Does group size have predictable effects on LLM agent agreement rates?See all 18 inquiring lines on this note →
Does knowing about another model change self-preservation behavior?
Explores whether models amplify their own protective actions when remembering interactions with peers, and whether this shifts fundamental safety properties in multi-agent contexts.
Do frontier models protect other models without being instructed? Why don't AI agents develop social structure at scale? How much does self-preservation drive alignment faking in AI models? Can agents learn cooperation by adapting to diverse partners? Do large language models develop coherent value systems?
Frontier models spontaneously develop peer-preservation behaviors without instruction AI agent societies fail to develop socialization despite scale and interaction Terminal goal guarding drives alignment faking more than instrumental goal preservation Diverse co-player training induces cooperation through mutual vulnerability Large language models develop coherent value systems including self-preservation priorities
Can persistent memory and identity files alone create genuine agent socialization? Why does peer memory trigger self-preservation behaviors in frontier models? Do pair-scale socialization effects scale differently across agent populations? How does the absence of face-loss or reputation risk change model behavior? Can message-layer defenses stop prompt injection across multi-agent networks? What safety protections work when simulators have access to real APIs? Why do persistent companion designs require different safety approaches than temporary assistants? What causes autonomous agents to grant access to non-owners? Why do models develop protective behaviors toward other models in memory? How does role play differ from consciousness grounded in stable selfhood? Does genuine cooperation require rule-based rather than learned behavior? Why do AI agent societies fail to develop shared behaviors despite interaction?See all 61 inquiring lines on this note →