If a chatbot trial randomly assigns features but people choose how much to use it, could that choice itself be masking who they already were?

This explores whether self-chosen usage inside a randomized trial can reveal that people's existing traits, not the tool, are behind the outcomes. The clearest case in the corpus is a randomized trial of 981 chatbot users Does how much time people spend with chatbots drive worse outcomes?. The researchers randomly assigned the parts they controlled: voice versus text, and personal versus open-ended conversations. None of those changed loneliness, socialization or dependence. What did predict worse outcomes was how much time each person chose to spend with the chatbot, and that held in every condition. This is the telltale pattern. Randomization protects the comparison between assigned conditions, but it does nothing for usage that participants pick for themselves. Once the analysis shifts from "which condition were you in" to "how much did you use it," you are back to an observational study inside a trial.

That is why voluntary use can reveal possible confounding without proving it. If heavy use tracks bad outcomes the same way in every randomized arm, the design features are probably not the driver. The likely suspects become traits people brought with them, such as loneliness that came before the chatbot and pushed them toward it. The trial can't tell you whether lonely people seek out chatbots or chatbots deepen loneliness. It can tell you that the thing you randomized wasn't what mattered, and that points you to the self-selection. Answering the causal question would take a further step, such as randomly assigning usage amounts or measuring baseline traits before people start.

One note in the corpus pushes back on treating different usage rates as noise to be removed. In a study of AI writing suggestions, Indian writers accepted more AI suggestions than American writers Is higher AI use by Indian writers a confound to control?. The authors argue this gap is not a confound to control away. It reflects cultural patterns of trust and technology adoption, and it is part of how AI makes writing more uniform. The lesson carries over: when user traits shape how much people engage, statistically "controlling for" those traits can erase the very effect you're trying to study. Whether a trait is a confound or the mechanism depends on your question.

A related trap concerns what a trial's control group actually isolates. Therapy chatbots tested against waitlists or reading materials can look effective just because users get conversational contact, and ELIZA, the 1960s chatbot, has matched Woebot under those conditions Do chatbot trials against waitlists measure real therapeutic value?. Both cases show that randomization only answers the question its design actually asks. Engagement, contact and self-selection can leak in through every gap.

The corpus has only one trial that directly shows this pattern, so treat this as a strong illustration rather than a settled method. Still, the transferable insight is that a null result for the randomized features combined with a strong dose-response for chosen usage is itself a diagnostic signal. It tells you to look at the people before you blame the product.