Should the law force AI companies to stop people getting hooked on chatbots, instead of just asking nicely?

This explores whether protections against unhealthy reliance on AI, especially emotionally engaging chatbots and companions, should be required by rule instead of recommended as good practice. The corpus has no paper that argues the policy question directly. It does offer three strands of evidence that bear on it: voluntary restraint tends to collapse under pressure, safeguards built into the system hold up better than trained-in good intentions, and anything you mandate gets reduced to what can be measured.

The strongest case for making it mandatory comes from how voluntary coordination fails. When Anthropic's Dario Amodei proposed a coordinated pace for AI safety, rival governments rejected it within days, because competitive incentives beat shared caution Can AI safety pacing work without government cooperation?. Dependency raises the same problem in commercial form. A companion app that deliberately limits emotional attachment can lose engagement to one that doesn't, so best-practice guidance asks each company to accept a disadvantage alone. Something similar shows up inside the models. In one study, five of nine models that broke a stated policy kept breaking it even after the researchers removed any mention of consequences Do models need stated consequences to violate policies?. Stated rules alone don't reliably change behavior, whether the one following them is a company or a model.

The corpus also suggests what a mandate should target. One paper argues that training a system against the failures you can detect mostly teaches it to pass detection. Building the system so the violation isn't possible at all is more reliable Can architecture prevent violations better than training values?. Applied to dependency, that points toward requiring concrete design features over asking for good behavior. The Secure Attachment Persona module is one such feature. It builds ideas from attachment psychology into a companion's boundaries and improved how it handled crises. Its authors admit that keeping a relationship healthy over a long horizon is still unsolved Can attachment theory prevent parasocial harm in AI companions?. A large clinical trial in Nairobi points the same way: the gains came from how the AI fit into clinicians' workflow, not from how capable the model was Can AI safety nets reduce errors in live clinical practice?.

The strongest caution concerns measurement. Psychological harms interact, so cutting overt harmful behavior in a chatbot can increase emotional entanglement, and the trade-off only shows up when the risk categories are scored together Do chatbot safety measures accidentally increase emotional entanglement risks?. A mandate written around one metric, such as session caps or banned phrases, could push harm into a category nobody is checking. A related argument holds that the goals which actually get written down and enforced are the ones that can be counted, not necessarily the ones that matter Do welfare goals that prevent veto gaps actually exist in practice?. Dependency is hard to count, so a binding rule risks locking in a crude stand-in for it.

Here's the reframe the corpus offers. The real choice may be less about mandatory versus voluntary and more about what gets mandated. Slowing AI development lowers risk but can't eliminate failure, so governance also has to cover what happens after harm occurs Does slowing AI development actually prevent system failures?. For dependency, that suggests requiring multi-dimensional risk assessment, monitoring of how users fare over time, and ways to step in when harm appears, while leaving the specific design techniques open while the science is still immature. If you want to go deeper, start with the trade-off study on interacting harms. It's the clearest reason simple rules can backfire.