AI assistants remember what you like, but seem to use that knowledge less in health and therapy chats than elsewhere.
Why do health and therapy preferences show weaker utilization than other preference types?
This explores why AI systems that know a user's preferences apply them less reliably in health and therapy conversations than in areas like shopping or travel, and whether the collection explains the gap.
This explores why AI systems that know a user's preferences apply them less reliably in health and therapy conversations than in other areas. No note in the collection measures a health-versus-other gap directly, so what follows is the closest evidence. The links between notes are my inference, not findings.
The starting point is a general gap between remembering and acting. In paired tests across 16 systems, agents passed recall tests but didn't reflect the preferences in their behavior, and most failures came when the agent interpreted the information, not when it retrieved it Why do LLM agents remember preferences but not act on them?. That fits health and therapy well. "I like window seats" is a rule an agent can apply directly. "Don't just tell me to exercise more" or "I need you to listen, not fix" has to be read out of context, and a lot rides on how it's read. Related work finds that abstract preference summaries beat retrieved past interactions Does abstract preference knowledge outperform specific interaction recall?. A health preference buried in an earlier emotional exchange is the kind of specific detail that is hard to turn into behavior.
The second pressure is that therapy is where a model's trained habits work hardest against what the user wants. When users share emotions, LLM therapists default to solution-focused advice, a hallmark of low-quality therapy, probably because RLHF rewards helpfulness Do LLM therapists respond to emotions like low-quality human therapists?. More broadly, LLMs produce 77.5% fewer grounding acts than humans (the small moves that establish shared understanding), and preference optimization makes this worse because fluent, confident answers get rewarded Does preference optimization damage conversational grounding in large language models?. A user who has said "just listen" is asking the model to resist the behavior it was tuned for. In a domain like product recommendations, the stored preference and the model's default rarely conflict.
A third factor may be that alignment training averages people out. One reward model fit to pooled preferences provably erases minority viewpoints Can a single reward model represent diverse human preferences?. Annotation responses also mix genuine preferences with ones people construct on the spot Do all annotation responses measure the same underlying thing?. Health and emotional preferences are highly individual, so an averaged "good therapist" style probably overrides them more often than an averaged "good restaurant" does.
The corpus also shows personalization working in therapy when it is an explicit, structured decision. CaiTI used Q-learning to pick which of 37 functioning dimensions to screen next, adapting to patient history over 24 weeks, and therapists judged its choices as matching clinical intuition. The same note reports that GPT-4 tended to echo users' feelings instead of giving objective guidance, while Llama-based models avoided this in structured CBT tasks Can reinforcement learning personalize which mental health areas to screen?. So the weakness looks less like health preferences being hard to store and more like their being hard to apply in free-form conversation. Checking whether the original benchmark breaks results down by domain would confirm or refute this reading.
Sources 7 notes
Paired Know and Act tests across 16 systems revealed a large gap: agents pass recall tests but fail to reflect preferences in behavior. Comprehension failures during interpretation dominate over retrieval failures, suggesting the bottleneck lies in applying stored information rather than retrieving it.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
Using the BOLT framework, researchers found LLMs offer solution-focused advice during emotional disclosure—a hallmark of low-quality therapy—yet also reflect more on client needs and strengths than typical poor human therapy, creating an unusual hybrid profile likely driven by RLHF's helpfulness bias.
Research shows LLMs generate 77.5% fewer grounding acts than humans, and RLHF preference optimization actively worsens this gap. The optimization target—fluent, confident responses—directly undermines the communicative work of establishing shared understanding.
MaxMin-RLHF proves an impossibility result: fitting one reward model to aggregated preferences silently erases minority viewpoints. The solution is learning a mixture of preference distributions and optimizing a MaxMin objective from social choice theory to protect the worst-off groups.
Show all 7 sources
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
CaiTI's Q-learning system adaptively selected which of 37 functioning dimensions to screen next based on patient responses over 24 weeks, validated by therapists as matching clinical intuition. However, GPT-4 models interpolated user feelings rather than providing objective guidance, a limitation Llama-based models avoided in structured CBT tasks.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Measuring Human Preferences in RLHF is a Social Science Problem
- A Computational Framework for Behavioral Assessment of LLM Therapists
- Comparing Human and AI Therapists in Behavioral Activation for Depression: Cross-Sectional Questionnaire Study
- Capturing Individual Human Preferences with Reward Features
- Beyond Preferences in AI Alignment
- Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment