Does showing a chatbot's inferred user model increase user control?

TalkTuner is an end-to-end prototype built on LLaMa2Chat-13B that uses linear probes trained on the model's internal activations to extract an approximate "user model" — representations of a conversational partner's age, gender, education level, and socioeconomic status (SES). The researchers paired this with a dashboard that displays the inferred user model alongside the ongoing chat in real time and lets the person edit it directly, rather than through prompting. The abstract reports that users "appreciate seeing internal states, which helped them expose biased behavior and increased their sense of control." In the study, seven participants described a stronger sense of transparency (P4: "[the dashboard] makes it more transparent how the model is and how that could be feeding into its responses"), and nine called the view "engaging and interesting." But five participants called the inference itself "uncomfortable" — P16: "there's an uncomfortable element to think that AI is analyzing who I am behind the screen" — and seven raised privacy concerns, worried the inferred demographics could feed targeted advertising.

The paper's own explanation for why exposure plus control worked is that the user model existed either way, invisibly, before the dashboard; making it visible just moved the choice from the system to the person. P8 captures this directly: "if it [the user model] was always there, I'd rather see it and be able to adjust it, than having it be invisible." Five participants spontaneously compared dashboard control to prompt engineering and preferred it — P17: "I feel very strongly about not having to type a super long prompt with all my information over and over again" — framing the control panel as a shortcut around having to re-assert context in every message.

This sits closely alongside Does revealing an AI's strategies help or harm user agency?: both studies expose a chatbot's hidden internal state and find the reaction splits between empowerment and discomfort, though TalkTuner exposes inferred demographic attributes rather than conversational strategies. It also extends Do users notice when LLMs infer and retain private information? — TalkTuner's participants likewise had no idea the inference existed until the dashboard showed it, and then wanted to correct it. The dual reaction also echoes Does chatbot personalization build trust or expose privacy risks?, with the same combination of engagement and privacy concern showing up around a different mechanism (direct activation probing rather than conversational memory). A companion note on this same source, Can steering a chatbot's user model change its factual claims?, describes the specific risk the dashboard was built to counter.

The study drew on a small, highly educated participant pool using one open-source 13B model, and the paper is explicit that this limits the findings: "Continuing to experiment with a broader sample, perhaps through public deployment of prototype systems, would be important for understanding the full design picture." The probes themselves are also only "approximately correct," by the authors' own description, not a verified ground truth of what the model represents. What the excerpt supports is that, for this prototype and this sample, surfacing and allowing control over an inferred user model changed people's sense of agency and trust — not that any particular dashboard design, or probe technique, would generalize to production chatbots or broader populations.