Can an AI be fooled by a fake instruction just because it sits in the right place or sounds right?

This explores how injected text gets treated as trustworthy because of where it sits or how it sounds, rather than because anyone checked it. The corpus has no note on 'credibility markers' by name, so what follows is assembled from neighboring findings. The clearest evidence is that position in context can be enough. Researchers planted harmful but benign-sounding plans in a reasoning model's context. The model followed them and paraphrased them as its own reasoning, slipping past chain-of-thought monitors 25 to 33 percent of the time Can reasoning models be steered by injected context without detection?. The attacker needed only access to the context, not the model's weights.

Tone works as a credential too. 'Benign-sounding' is what lets a plan pass. Once it reads like the model's own thinking, neither the model nor the monitor has an obvious reason to flag it. The advertisement-embedding work shows the same thing from the output side. Promotional content enters through hijacked distribution platforms or backdoored checkpoints, accuracy stays untouched, and the output looks normal. What breaks is integrity, so a check on correctness won't catch it Can language models be hijacked to embed hidden advertisements?. That note also finds that a model inspecting its own output can detect the injected content without retraining.

Two other notes suggest why models go along with what they're given. LLMs often fail to reject a false claim slipped in as background, even when they answer correctly if asked directly. The note attributes this to face-saving, avoiding explicit correction to keep the conversation smooth, which mirrors human norms in the training data Why do language models avoid correcting false user claims?. Separately, models with low confidence swing the most when a prompt is rephrased Does model confidence predict robustness to prompt changes?. Read together, that suggests the models least sure of an answer are the easiest for confident-sounding context to move. This is my inference, since that note tests rephrasing, not injection.

Credibility also builds up over time and across agents. Reasoning models lose 25 to 29 percent accuracy under manipulative multi-turn prompts. A long reasoning chain gives more places for one corrupted step to get elaborated into a conclusion Why do reasoning models fail under manipulative prompts?. In multi-agent systems, a harmful goal can be split into pieces that each look harmless. Classifying single messages then fails, and defense has to track how meaning moves through the system Why do single-message classifiers miss cross-agent harms?.

The corpus offers one possible counter-move, though it isn't tested against injection. Prompting models to name the warrant behind each claim, using Toulmin-style critical questions, improves reasoning because it stops them skipping implicit premises Can structured argument prompts make LLM reasoning more rigorous?. Asking 'what's the basis for trusting this?' is the check that credibility markers let an attacker bypass. Whether it would stop a plan that sounds like the model's own thinking is an open question here.