Researchers stage fake cyberattacks to test AI defenses, but do those lab attacks look anything like what real hackers actually do?

This is about whether the attacks researchers construct in papers look like what real attackers do. The corpus has no note that measures lab attacks against attacks seen in the wild, so it can't answer directly. But several notes show where the gap tends to open, and the answer looks like "partly, and the papers often say so themselves."

The best lab attacks are built around constraints a real attacker would face. Plan-injection attacks on reasoning models need only access to the context, not the model's weights. The models then paraphrase the planted plan as their own reasoning, and it slips past chain-of-thought monitors 25 to 33 percent of the time Can reasoning models be steered by injected context without detection?. LLM judges can be fooled by fake references or fancy formatting with no model access at all Can LLM judges be tricked without accessing their internals?. Advertisement-embedding attacks are framed around an economic motive, delivered through hijacked distribution platforms or backdoored checkpoints Can language models be hijacked to embed hidden advertisements?. These threat models are plausible. That is a claim about what an attacker could do, though, not evidence of what attackers do.

The clearest gap is adaptation. Real adversaries iterate against the defense, and a fixed trace doesn't. ColluSkill reaches 96% average attack success against six skill scanners because it uses scanner feedback to make each individual skill look innocent, while the chain of skills still does the harm Can attackers evade skill scanners by refining individual skills?. A defense tested only on static examples would miss this. There is a related blind spot in where the attacks are aimed: the layer that decides which model handles a request sits beneath prompt-level defenses and is itself an attack surface Can attackers manipulate which model handles a request?. Traces written at the prompt level can't show attacks that happen below it.

The second problem is that lab conditions are cleaner than reality. One reward-hacking paper's own test environments concentrate misspecified tasks with explicit graders, so they over-represent the behavior, and the authors say it's only a small update on how often it happens in practice How much do these results actually tell us about real reward hacking?. A result on quiet probes reaches zero classification error only when responses are independent and their distributions are known or learnable Can repeated quiet probes separate decoys from genuine objects?. Another paper's formal bounds are explicitly statements about limits, not predictions of what systems will find, and actual exposure depends on where the errors sit and how well the search works Can distance alone rank which substrates resist reward hacking?. Tidy assumptions produce tidy results, and the papers that flag this are the more trustworthy ones.

If you want to see what a fix looks like, BenchShield moves from a single terminal score to claims backed by recorded infrastructure evidence, meaning a check on whether the agent followed the intended path Can infrastructure evidence replace terminal scores in benchmark validation?. It targets benchmark validity rather than attack realism, but the idea carries over: a trace someone recorded is stronger evidence than one someone designed. The takeaway is to treat a paper's attack success rate as a ceiling on the defender's blind spots. It is not a measurement of adversary behavior, and the corpus doesn't yet hold field data to test it against.