When a model ignores what you tell it, the fix often lies inside its internal workings, not in better prompts.

This explores whether the opacity of what models learn internally turns into brittleness: failures nobody can predict, diagnose, or fix. One caveat up front: the collection has little that addresses interpretability head-on. What it does have points to an interesting answer. Learned representations do cause robustness problems. But you don't always have to read the internals to anticipate those problems, and when you do want to fix them, the representations themselves are often where you have to work.

The clearest case is a model ignoring what you tell it. When knowledge absorbed during training is strongly held, models produce answers that contradict the information right in front of them. Better prompting can't talk them out of it. Researchers had to intervene directly on the internal representations to make the context win Why do language models ignore information in their context?. That's the robustness cost of opacity in one picture: the problem sits in a layer that ordinary instructions can't reach. A similar pattern shows up in grammar. Models that handle everyday text fluently consistently misidentify embedded clauses and complex phrases, and they get worse as sentence structure gets deeper Why do large language models fail at complex linguistic tasks?. The same goes for ambiguous sentences, where GPT-4 sorts out the competing meanings only about a third as often as humans do Can language models recognize when text is deliberately ambiguous?. Both look like what happens when learned representations capture surface statistics rather than the underlying rules. Ordinary benchmarks hide that gap until someone tests for it.

The surprising counterpoint is that you can sometimes predict failures without opening the black box at all. One line of work treats an LLM simply as a machine trained to predict the next most probable word. From that description alone, researchers correctly predicted that tasks with improbable-looking answers, like reciting the alphabet backwards or counting letters, would trip models up even though they're logically trivial Can we predict where language models will fail?. Confidence works the same way from the outside. Instead of trusting what a model 'feels' internally, one method looks up how the model did in the past on cases where it was similarly confident, and that matches far more expensive checks Can past performance predict when a model will be right?. Uninterpretable internals don't leave us blind. They push us toward reasoning about behavior.

The twist is that not every strange internal change is a failure. When models face unfamiliar, out-of-distribution tasks, their internal activity becomes markedly sparser. You might read that as the model breaking down, but the evidence suggests it's adaptive filtering that helps hold performance steady Do language models sparsify their activations under difficult tasks?. So the robustness problem isn't simply that representations are hard to read. It's that without reading them, we can't tell a coping mechanism from a breakdown. If you want to see where robustness gets designed in rather than discovered afterward, the work on teaching models to abstain when uncertain is a good next stop Can models learn to abstain when uncertain about predictions?.