Randomized AI trials can miss the users who'd gain most, through who gets recruited, what they're compared against, and averages that bury them.

This explores how randomized trials of AI tools can miss the people who would gain the most. That can happen through who gets recruited, what the tool is compared against, or how averaging buries the subgroups where the real effect lives. The corpus has no study built around this question. It does contain a cluster of notes that, read together, show three different ways a trial can look past its most important users.

The first is the comparison. A trial can be randomized and still measure the wrong thing. Therapeutic chatbot studies often compare the bot against a waitlist or a psychoeducation handout, and Do chatbot trials against waitlists measure real therapeutic value? argues that this design mostly measures the value of having someone, or something, to talk to. The tell is that ELIZA, a 1960s pattern-matching script, performed about as well as Woebot. A design like this can't tell you who the therapy-specific benefit reaches, because it never isolates that benefit. The people who would gain most from the actual therapeutic mechanism get counted together with everyone who just liked having contact.

The second is averaging. Most trials report a mean effect, and means are set by typical users. The persona-simulation research makes this concrete. Should persona simulation prioritize coverage over statistical matching? finds that matching the statistical shape of a population misses rare but consequential user configurations, and that you need to deliberately cover the edges. Can simulated users reveal what offline benchmarks miss? makes the same point about evaluation in general: outcome-only scoring hides how different people phrase requests and judge results. If the users who would benefit most are unusual, such as novices, people working on hard cases, or people with atypical needs, a well-powered average can still say 'modest effect' while their gain disappears into it. Can AI guidance reduce anchoring bias better than AI decisions? hints at where those gains tend to cluster: AI help that points people toward what matters does the most on the hard cases people would get wrong alone. A trial dominated by easy cases would barely register that.

The third is the outcome measure. Even with the right people enrolled, a trial can measure the wrong thing. Does AI assistance always help reasoning or does it carry hidden costs? shows that AI suggestions can be correct and still hurt performance by breaking concentration. Does supervised fine-tuning improve reasoning or just answers? shows that final-answer accuracy can go up while the reasoning behind it gets worse. The trial lesson carries over: if you score only the final output, you can miss both hidden costs to experts whose focus gets broken and hidden gains to learners whose process improves.

Here's the twist you might not expect. Simulated users are increasingly proposed as a way to pre-screen populations before running expensive trials, and they are least reliable exactly where this question points. Can AI personas reliably replicate human experiment results? finds that AI personas reproduced most strong published effects but were unreliable on marginal ones, with both false positives and false negatives. Subgroup benefits are usually the marginal, weaker-signal effects. So the tool that promises to find overlooked users is weakest at finding them, unless it is designed for coverage rather than averages. One more gap: workplace field studies that break results down by experience level would fit this question directly, but they weren't in this retrieval.