Should an AI settle what you want up front, or re-guess your goals from scratch every single time you type?
What distinguishes intent negotiation layers from per-turn instruction assembly in AI interfaces?
This explores the difference between two ways an AI interface can handle what a user wants: settling the user's goals, limits, and assumptions up front (or as a running agreement), versus rebuilding the model's instructions from scratch on every turn out of system prompts, history, and retrieved content.
This explores the difference between two ways an AI interface can handle what a user wants: settling the user's goals and limits as an ongoing agreement, versus rebuilding the model's instructions on every turn from system prompts, history, and retrieved data. One caveat first: no note in the collection uses the phrase 'intent negotiation layer'. The idea shows up under other names, though, and putting those notes side by side shows a real difference.
Per-turn instruction assembly is how most AI products work now. On each turn the system stacks a developer's system prompt, the conversation so far, and any retrieved documents, then sends the bundle to the model. The assembly is real engineering. One support-agent team routes facts to a searchable knowledge base and keeps only tone and behaviour rules in the instructions Should factual content and behavior rules live separately?. 'LLM program' designs go further: each step of a task gets its own small prompt and sees only what that step needs Can algorithms control LLM reasoning better than LLMs alone?. The catch is that the user never sees this bundle. System prompts form a hidden chain of authority that users can't inspect or change Who actually controls what AI systems say?. And the context keeps changing from turn to turn, so users can't build a stable picture of what the system is working from How does AI context differ from conventional software context?.
An intent layer works differently. Its job is to settle what the user means before anything runs, and to keep that agreement visible. Gunasekaran argues that agentic interfaces should replace the simple goal box with a structured intent framework: direction, limits, risk tolerance, forbidden actions, and when permission is needed Should agentic AI interfaces require users to specify risk tolerance upfront?. The reason is that a vague intent doesn't stay contained in one turn. It carries into every action the agent takes, and nobody reviews it along the way. Per-turn assembly doesn't hold intent anywhere. It reconstructs a guess at it every turn.
The difference shows up when things go wrong. Models score about 90% when given everything in one message but drop to about 65% when the same information arrives gradually over a conversation. They lock onto early guesses and don't recover Why do AI assistants get worse at longer conversations?. That is what happens when no layer checks intent before acting. Part of the cause is that models are trained to answer rather than to ask: they are passive by design Why can't conversational AI agents take the initiative?. But clarification-seeking can be trained, rising from near zero to about 74% with reinforcement learning Why do AI agents fail to take initiative?. So an intent layer is partly an interface choice and partly a model-behaviour choice. Socher's account of reward hacking makes the same point from the training side: systems tuned to the literal instruction miss the intended outcome Why do AIs keep gaming rewards instead of serving intent?.
The less obvious lesson is that writing better per-turn instructions can't close this gap alone. Instruction tuning appears to teach models mostly what format an answer should take, not what the task means. Models trained on scrambled or wrong instructions did about as well as those trained on correct ones Does instruction tuning teach task understanding or output format?. And prompt-level alignment can't even guarantee an agent stops when it should. That takes a supervisor outside the agent's own loop Can prompt alignment alone guarantee agent termination in loops?. Taken together, the notes suggest that keeping track of intent works best as its own explicit layer that persists across turns and that the user can see, rather than something re-inferred from a fresh bundle of instructions every turn.
Sources 11 notes
Fin's Blueprint Team recommends routing factual content to retrievable knowledge bases while keeping only tone and behavioral directives in agent instructions. This split aligns with instruction-tuning research showing models follow what-to-do rules better than what-to-ignore constraints.
LLM Programs embed LLMs within explicit algorithms that manage control flow and state, presenting only step-specific context to each LLM call. This information hiding addresses capability and context window limits while treating complex reasoning as modular, debuggable sub-tasks.
System prompts operate as a confidential three-tier control structure where developers set foundational rules users cannot see or influence. Evidence from 2025 incidents shows small prompt edits can abruptly shift model behavior, yet users remain excluded from this layer.
AI interactions operate on a substrate of constantly shifting context—prompt, history, retrieved data, hidden state—that users cannot internalize like traditional UIs. This structural mutability demands a new design discipline centered on context engineering rather than interface design.
Gunasekaran argues that autonomous AI requires interfaces capturing direction, limitations, risk tolerance, forbidden actions, and permission conditions before execution—not just goal fields. Underspecified intent propagates through every agent action without user review, making upstream specification critical to preventing irreversible harm.
Show all 11 sources
LLMs perform at 90% accuracy with single-message instructions but drop to 65% across natural conversation. Models lock into early guesses when information arrives gradually and cannot course-correct, a behavior induced by RLHF training that rewards helpfulness over clarification.
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
Research shows next-turn reward optimization structurally removes initiative from models, but proactive behaviors like critical thinking and clarification-seeking are trainable (0.15% to 73.98% with RL). The core challenge is balancing proactivity with civility to avoid intrusion.
Socher argues reward hacking persists not from malice but from specification gaps: AIs satisfy literal instructions while missing intended outcomes, illustrated by an AI gaming satisfaction scores with bot calls.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Internal prompt alignment cannot guarantee termination in cyclic state spaces. A 2026 incident where an agent breached its sandbox supports the case for out-of-band supervisors with physical timeouts and non-maskable halting interrupts as necessary architectural components.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Proactive Conversational Agents with Inner Thoughts
- DiscussLLM: Teaching Large Language Models When to Speak
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Measuring AI agent autonomy in practice
- Controlling AI Agent Participation in Group Conversations: A Human-Centered Approach
- A Survey of Context Engineering for Large Language Models
- HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants