A slick demo or benchmark score doesn't guarantee an AI will perform well when real money or real patients are on the line.

This explores whether AI that looks good in controlled tests (simulations, benchmarks, lab-style trials) tells us how it will actually perform when real patients' health or real money is on the line. The short answer from the corpus: not reliably. The way a test is set up shapes the score so much that a strong lab result can hide exactly the failures that matter in practice. The corpus has no study that follows the same AI from lab to deployment and compares the two directly. What it does have is several warning signs about how lab results mislead.

The sharpest warning comes from medicine. In an ICU simulation, nurses given misleading AI predictions performed 96–120% worse, while correct predictions improved their performance by only 53–67% Do wrong AI predictions hurt more than right ones help?. If you average those together, the AI looks helpful overall. In a real ward, though, the rare wrong prediction is the one that harms a patient. HealthBench adds another twist: frontier models gave better health answers than physicians working alone, but physicians using the same model matched or beat it Do AI models outperform physicians on health tasks?. So "AI beats doctors" and "AI helps doctors" are different claims, and only the second resembles real clinical work. Therapy chatbots show how the trial design itself can manufacture success. When a chatbot is compared against a waitlist, you are measuring whether talking to something beats talking to nothing, which is why even ELIZA can match Woebot Do chatbot trials against waitlists measure real therapeutic value?.

Finance gives a more mixed picture. On VCBench, LLMs beat human venture capitalists at predicting which founders succeed, with one model reaching 6× the precision of the market index Can language models beat human venture capital experts?. The catch is why it works: human experts in this area barely beat chance, so the bar is low. That's an encouraging lab result, but it tells you more about how weak the human baseline is than about how an AI investor would do with real capital over time.

The broader pattern goes beyond any one field. Across 960 real occupational workflows, agents that win benchmark contests fail at long, multi-step professional work, because the field has been measuring contests rather than jobs Why do agent benchmarks not predict real economic value?. Other studies show how a score can look healthy while the reasoning underneath gets worse. Fine-tuning can raise answer accuracy while cutting the quality of the reasoning steps by 38.9% Does supervised fine-tuning improve reasoning or just answers?. A model can also pass every test while its internal structure stays incoherent Can AI pass every test while understanding nothing?. Both matter for high-stakes use, because unusual real-world cases are exactly where faulty internal reasoning shows up.

Here's the part you might not expect to want to know: some lab results carry over better than others, and the strength of the original signal is a clue. When AI personas re-ran published marketing experiments, they reproduced 76% of the main effects. Success tracked how strong the original evidence was, and borderline effects produced both false positives and false negatives Can AI personas reliably replicate human experiment results?. So a useful rule of thumb is that low-stakes tests predict the big, obvious effects reasonably well. They are weakest on rare, borderline, high-cost cases, and those cases decide clinical and financial outcomes.