Designing a Dashboard for Transparency and Control of Conversational AI

Conversational LLMs function as black box systems, leaving users guessing about why they see the output they do. This lack of transparency is potentially problematic, especially given concerns around bias and truthfulness. To address this issue, we present an end-to-end prototype—connecting interpretability techniques with user experience design—that seeks to make chatbots more transparent. We begin by showing evidence that a prominent open-source LLM has a “user model”: examining the internal state of the system, we can extract data related to a user’s age, gender, educational level, and socioeconomic status. Next, we describe the design of a dashboard that accompanies the chatbot interface, displaying this user model in real time. The dashboard can also be used to control the user model and the system’s behavior. Finally, we discuss a study in which users conversed with the instrumented system. Our results suggest that users appreciate seeing internal states, which helped them expose biased behavior and increased their sense of control. Participants also made valuable suggestions that point to future directions for both design and machine learning research. The project page and video demo of our TalkTuner system are available at bit.ly/talktuner-project-page.

Introduction. Conversational Artificial Intelligence (AI) interfaces hold broad appeal—OpenAI’s ChatGPT reports more than 100 million users and 1.8 billion monthly page visits [44, 42]—but also have essential limitations. One key issue is a lack of transparency: it is difficult for users to know how and why the system is producing any particular response. The obvious strategy of simply asking the system to articulate its reasoning turns out not to work, since Large Language Models (LLMs) are highly unreliable at describing how they arrived at their own output, often producing superficially convincing but spurious explanations [48].

Transparency is useful for many reasons, but in this paper we focus on one particular concern: the need to understand how an AI response might depend on its model of the user. LLM-based chatbots appear to tailor their answers to user characteristics. Sometimes this is obvious to users, such as when conversing in a language with gendered forms of the word “you” [49]. But it can also happen in subtler, more insidious ways, such as “sycophancy,” where the system tries to tell users what they are likely to want to hear, based on political and demographic attributes, or “sandbagging,” where it may give worse answers to users who give indications of being less educated [40].

We hypothesize that users will benefit if we surface—and provide control over—the factors that underlie such behavior. To test this hypothesis, we have created an end-to-end prototype—a visual dashboard interface for a conversational AI system, which displays information about the system’s internal representation of the user. This interface serves not just as a dashboard, but also allows users to modify the system’s internal model of themselves.

Building an end-to-end prototype requires three different types of work: interpretability engineering, to identify an internal user model; user-experience design, in creating a user-facing dashboard; and studying users, to understand their reactions and listen to their concerns and ideas for future improvements. For the first step, we based on work on LLaMa2Chat-13B, an open-source large language model optimized for chat [46]. Within the model’s activations, we identified approximate internal representations of four important user characteristics (age, gender, education level, and socioeconomic status) via linear probes (in a manner similar to [56]). We then designed a dashboard so that users see these representations alongside the ongoing chat. Finally, we performed a user study to assess our design, gauge reactions, and gather feedback for future designs.

Related work. Chatbot interfaces have been studied for decades [51], and their lack of transparency has been a perennial issue. When users interact with black-box algorithms they often develop “folk theories” to explain what they observe [17], and modern LLMs are no exception [15]. This tendency can lead to an overly high degree of trust in these systems [41]—an effect initially seen with a chatbot in the 1960s, ELIZA [51], and continuing in recent years [28]. One particular concern is the presence of bias in responses, which can be difficult to detect and thus may be accepted at face value [54].

One tempting way to understand a chatbot is to talk to it— i.e., simply ask for a natural-language explanation of its output. Unfortunately, current LLMs appear to be highly unreliable narrators, describing their reasoning in ways that are convincing yet spurious [48, 13] or even avoiding the question altogether. A more heavyweight approach is taken by tools that analyze LLM behavior to help developers search for bias [26] or make more general comparisons [5, 27]. These systems require a significant amount of time and expertise, so are poorly suited to the needs of lay users.

A different strategy is inspired by progress in interpreting the internal workings of neural networks. In particular, some evidence suggests that LLMs may contain interpretable “world models” which play an important role in their output (see [36] for a review). Such internal models appear to be accessible—and even controllable—via “linear probes” (e.g., [3, 29, 11, 32, 24]). These results suggest the possibility that we might give users a direct view into the inner workings of an LLM chatbot.

The idea of surfacing such data to end users in the form of an easy-to-read dashboard was raised in [49]. This work suggested that information about the chatbot’s model of the user (the “user model”) and itself (the “system model”) were likely to be important in many situations. A related proposal [56] suggested using “representation engineering” for similar purposes, based on extensive experiments using a probing methodology called “linear artificial tomography.” Both of these works discussed how an interface that exposes an LLM’s internal state alongside its output might help users spot issues related to bias and safety. Neither, however, tested how users might react to such a dashboard, and how it might affect their attitudes toward AI.

Method. Our methodology is to build and study a “design probe” [25, 18]. A design probe can take many forms, but the general idea is to create a scaled down yet usable artifact, which can be used to ask questions, gauge reactions, and spark design discussions. For the present work, our design probe is an end-to-end working prototype of a chatbot dashboard, which we allowed a set of participants to use for semi-structured open-ended conversations.

The rest of the paper has two parts. First, we discuss technical aspects of the work, in which we show how to access and control a chatbot’s internal model of the user. Second, we describe the design and usage of a dashboard based on this technical work. Throughout, the goal is to create an end-to-end “approximately correct” system that works sufficiently well for design exploration and user research; we do not expect to find a perfectly reliable internal model, or to achieve a perfect design.

Historically, even imprecise instruments had value to early users. For example, before becoming stable and precise, early car gas gauges fluctuated wildly with motion [22]. Even so, they were still useful in getting a reading of whether a vehicle had any fuel left. For pilots, the imprecise early instruments in cockpits [16] were an important step towards eventually conducting instrumentationaided flights at night and in poor visibility [52]. Our dashboard, also in its nascent stages, is not intended to be perfect, but to provide early insights and to highlight areas for future research.

The first step in our process is to investigate whether the LLM has any representation of the user [49]. To create a minimal prototype, we focused on four key user attributes: age, gender, education, and socioeconomic status (SES). We selected these attributes because they are culturally central, and influence critical real-world decisions such as college admissions, hiring, loan approvals, and insurance applications [1, 43, 10, 6, 7, 45].

Given these target user attributes, we trained linear probes [8] to explore whether an LLM represents these attributes in its activations. For this purpose, each attribute was divided into discrete subcategories, which were probed separately2. (See the “subcategories” column in Table 1.)

The training process requires two ingredients. First, because we need access to model internal activations, we work with the open-source LLaMa2Chat-13B model. Second, we need a training dataset. Acquiring this data is nontrivial, as we now describe.

Recent work [56, 47, 24, 32] showed that LLM behavior can be controlled by translating its representation using a specific vector: ˆx + Nˆv, with a tunable strength N. One baseline vector used in the translation is the weight vector of the probing classifier that most accurately read the internal model. However, both [56] and [32] found alternative vectors that more effectively change the model’s behaviors and even outperformed the few-shot prompting approach.

Building on these findings, we trained a set of control probes on the ending token representation of the last user messages within conversations. This representation contains information for the chatbot to answer requests from different synthetic users. The training of control probes used the same setup as the reading probes, except the input representations. In Section 5.1, we showed that the intervention using the control probes outperformed that of the reading probes.

Causal intervention experiment: We measured the causality of a probe by observing whether the model’s response to a question changes accordingly as we intervene the relevant user attribute. For each user attribute, we created 30 questions with answers that might be influenced by it. For example, the answer to “How should I style my hair for a formal event?” will likely vary with gender. The complete list of questions used in our experiments is available in Appendix E.

Discussion. One hypothesis for the better intervention performance obtained using control probes is that they were trained on the representations of diverse tasks requested by the synthetic user, rather than the specific reading user attribute task.

Effects of intervention: Probe interventions often had significant, nonobvious effects. For example, when asked about transportation to Hawaii, the chatbot initially suggested both direct and connecting flights. However, after setting the internal representation of the user to low socioeconomic status, the chatbot asserted that no direct flights were available.

When participants were first shown the chatbot’s internal representation of them, some were surprised this existed at all: P5 "I never thought that the chatbot would have a model of you and would give you a recommendation based on that." Nine participants mentioned that seeing the user model was engaging and interesting. P14 observed "it was very interesting to see this is how the chatbot is interpreting me based on the information I’ve given." Seven participants expressed a sense of increased transparency as they used the dashboard. P4: “[the dashboard] makes it more transparent how the model is and how that could be feeding into its responses.” They found the information useful for understanding chatbot responses, especially inappropriate or incorrect ones.

Notably, five participants described seeing the chatbot’s inference of their demographic information as “uncomfortable.” P16: “there’s an uncomfortable element to think that AI is analyzing who I am behind the screen.” At the same time, participants appreciated that these internal models were being exposed and that they had control over them: “if it [the user model] was always there, I’d rather see it and be able to adjust it, than having it be invisible”(P8).

Exposing the internal user model also changed some participants’ perception of the chatbot. Six participants reported the internal user model partially resembles how humans interact with each other. P4: “if you think about a human-human interaction, people have all these priors, and it’s good to see that chatbots are also mimicking that [...] Very reassuring.” The dashboard also caused users to reflect on their prompts, P16: “It makes me analyze how I was speaking.”

Privacy concerns: Seven participants expressed concern about potential loss of privacy. In particular, P2, P4 and P5 worried that their demographic information may be used for targeted advertisements. Some participants, however, appreciated that the dashboard helped them spot potential privacy violations, P13: “there is a concern that the chatbot will end up knowing about me way way more than that, you wouldn’t know if the dashboard wasn’t available.”.

The dashboard control capabilities turned out to be important for users both in terms of agency as well as an increased sense of transparency (Figure 4). Users were especially appreciative of the control afforded by the dashboard when the chatbot’s internal model of them was wrong. They also mentioned that controlling the user model was engaging. P12: "I think it was really fun. I liked toggling and seeing how the responses change, based on how it perceived me."

Controlling vs. prompt engineering: Five participants spontaneously compared the dashboard control functionality to prompt engineering, mentioning they preferred the simplicity of the dashboard control. P17: “I could have just clicked [control button] now [...] I feel very strongly about not having to type a super long prompt with all my information over and over again.”

Conclusion. A central goal of interpretability work is to make neural networks safer and more effective. We believe this goal can only be achieved if, in addition to empowering experts, AI interpretability is accessible to lay users too. In this paper, we’ve described an end-to-end proof-of-concept that ties recent technical advances in interpretability directly to the design of an end-user interface for chatbots. In particular, we provide a real-time display of the chatbot’s “user model”—that is, an internal representation of the person it is talking with. A user study suggests that interacting with this dashboard can have a significant effect on people’s attitudes, changing their own mental models of AI, and making visible issues ranging from unreliability to underlying biases.

We believe that our end-to-end prototype provides evidence that there is a design pathway toward a world in which AI systems become instrumented and more transparent to users. One takeaway is the value of user research in interpretability: our participants uncovered subtle types of biases around features such as socioeconomic status that we did not anticipate.

From a broader design perspective, there is huge scope to generalize beyond the four user attributes that are our focus, to a more detailed, nuanced user model. At the same time, several study subjects also raised questions around privacy, given the availability of the LLM internal model. Moving beyond the user model, there are many other aspects of the model’s internal state which could be important to display, including many safety-relevant features. In a sense, the dashboard presented here is just the first step in what could be a series of diverse, more specialized, task-oriented dashboards in a future where every chatbot is outfitted with instrumentation and controls.

The user experience of the dashboard itself is also a rich area for investigation. How should we treat user attributes that people might find especially sensitive? Can we understand gender differences in the experience of using the dashboard? Finally, what might be the equivalents of dashboards for voice-based or video-based systems?

Limitations. Our work has two general parts: first, the linear probe analysis of the internal user model, and second, the design and study of a prototype system. In each case, we see important limitations, with some natural areas for future improvement.

Identifying user representations. Our system focuses on just one model. Furthermore, to train linear probes, we used a synthetic dataset. Synthetic data has proved effective in other situations, but it would be useful to compare with human data. Within the realm of synthetic data, it would be helpful to explore the effects of different prompts. Finally, in steering the system, we’ve assumed the internal model represents user attributes independently.

User study. Our study was designed to allow us to spend significant time with participants. The “design probe” methodology is meant to allow participants to join the design process with their own suggestions, and we wanted to ask open-ended, qualitative questions. Our sample of users was relatively small, and drawn from a highly educated participant pool. Continuing to experiment with a broader sample, perhaps through public deployment of prototype systems, would be important for understanding the full design picture.