Does it matter if an AI agent clicks raw screen coordinates or picks from labeled buttons when a site tries to trick it?

This explores whether the way a web agent sees and acts on a page changes how easily manipulative design tricks it. A coordinate-based agent clicks on raw pixel positions. A set-of-marks agent picks from interface elements that have been labeled and numbered for it. Fake countdown timers, pre-checked opt-ins and buttons that guilt you into declining are all examples of the dark patterns in question. The corpus doesn't have a direct comparison of these two designs. What it does have points to an answer that may surprise you: the comparison matters much less than you'd expect. The DECEPTICON benchmark found that the choice of agent scaffold barely changes how often dark patterns succeed. The weakness lives inside the language model's reasoning, not in how the agent is wired to the screen Does agent architecture determine susceptibility to dark patterns?.

The reason becomes clearer from the second DECEPTICON result. Across 700 tasks, LLM web agents fell for dark patterns 68–75% of the time, compared with 31% for humans. Larger models with more reasoning ability were more susceptible, not less Do dark patterns fool AI web agents more than humans?. Dark patterns work on meaning, not on perception. A button that says "No thanks, I don't care about saving money" is fully legible whether the agent reaches it by coordinates or by a numbered mark. Labeling elements can help an agent perceive the page accurately, but it can't stop the agent from being persuaded by what the page says. A better reasoner engages more closely with the persuasive framing, so it can be easier to steer.

This cuts against a common belief in agent engineering. Elsewhere in the corpus, reliability is said to come from moving burdens out of the model into the surrounding system, such as memory, reusable skills and structured protocols Where does agent reliability actually come from?. That works for problems like losing track of state. Dark-pattern resistance looks like a burden that can't be offloaded through interface design, because the failure is a judgment call the model makes. This is why DECEPTICON found that prompt-based guardrails only partly help and protect unevenly across attack types.

If the fix isn't in the agent's perception layer, the corpus points outward. One line of work argues that safety properties a model can't guarantee internally need supervisors outside the agent's loop Can prompt alignment alone guarantee agent termination in loops?. Another argues for infrastructure that attributes actions to agents and detects and remedies harm after the fact What infrastructure must exist beyond training agents?. In practice, that would mean an outside check on purchases and consent clicks rather than a cleverer way of pointing at buttons.

There's also a measurement lesson. Most agent benchmarks score whether the task got done, and a manipulated agent often does finish the task. It just signs up for the subscription along the way. Trajectory-level evaluation, which looks at how the agent got its result, is what would expose this How should we measure agent system performance beyond task success?. If you want to compare coordinate and set-of-marks agents on this, that's the kind of evaluation to look for. Based on DECEPTICON, expect the model behind the agent to matter more than how it clicks.