Do token probability distributions in LLMs track human reaction time patterns?
This explores whether the probabilities an LLM assigns to its next token line up with how long humans take to respond — the idea that low-probability outputs are 'harder' for a model the way slow reactions mark difficulty for people.
This explores whether token probabilities behave like a difficulty signal that mirrors human reaction times. No note in the collection directly measures reaction-time latency against next-token probability, so the honest answer is that the corpus addresses the surrounding territory rather than the literal claim — but the surrounding territory is unexpectedly rich, and it points both ways. The strongest 'yes-ish' thread is the 'embers of autoregression' work, which treats an LLM as a probability machine and predicts where it will fail: tasks whose correct answer is a low-probability sequence (reciting the alphabet backwards, counting letters) get systematically harder even when they're logically trivial Can we predict where language models will fail?. That's the same shape as the human reaction-time story — rare, effortful responses cost more — except the model pays in accuracy rather than milliseconds. The probability distribution does encode a graded notion of 'effort,' which is the premise your question rests on.
Where it gets interesting is that the collection also shows LLMs capturing human cognition directly. Models fine-tuned on psychology-experiment data out-predict purpose-built cognitive models at forecasting human decisions, and they fold individual differences into their embeddings Can language models learn to model human decision making?. So the link between LLM internals and human behavioral data isn't hypothetical — it's been demonstrated for choices. Reaction time would be the natural next variable to test, and the fact that decision-prediction already works is the reason someone would expect token probabilities to track latency too.
The sharpest counterpoint comes from a note arguing the analogy breaks at the level of time itself. LLM generation is sequential but atemporal: tokens are selected by probability with no intervening pause, revision, or 'duration in reflection' Does AI text generation unfold through temporal reflection?. Human reaction time is precisely a measure of that reflective duration — the thinking-time that changes what comes next. If you take this seriously, a high-probability and a low-probability token are emitted in the same computational beat; any correlation with human RT would be a correlation of *difficulty rankings*, not of *process*. The model isn't slower on hard tokens; it's just less certain.
A useful refinement sits in the work on high-entropy 'forking' tokens: only about 20% of tokens carry real decision-weight, the pivotal branch points where the distribution is genuinely uncertain Do high-entropy tokens drive reasoning model improvements?. That's a more precise candidate for what might map to human hesitation than raw probability across all tokens — most tokens are near-deterministic filler that no human would deliberate over either. And the theory-of-mind note is the cautionary footnote: where LLMs look like they're doing human cognition, they often default to surface pattern-matching rather than the underlying mental process Do large language models genuinely simulate mental states?. The thing you'd want to take away: token probability plausibly tracks *which* outputs are hard, and the corpus shows that ranking transfers to human-difficulty in decision tasks — but 'reaction time' specifically asks about temporal effort, and the most direct note in the collection says that's exactly the dimension LLMs don't have.
Sources 5 notes
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Token ordering in LLMs follows probabilistic selection without intervening reflection or revision. Human discourse gains meaning from temporal structure—time spent thinking changes what comes next—but AI text production lacks this duration-in-reflection despite appearing sequentially composed.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.