Does therapy environment difficulty calibration affect RL policy learning quality?

This explores two things at once — whether the difficulty of training problems shapes how well an RL policy learns, and whether that lesson maps onto therapy RL systems where the 'environment' is a live clinical conversation. The corpus speaks clearly to the first half and only obliquely to the second, which is itself the discovery here.

On difficulty calibration in general, the evidence is blunt: too-hard problems actively damage policies. Training on near-impossible samples teaches models to chase degenerate shortcuts — answer repetition, computation-skipping — and those shortcuts then leak backward and corrupt skills the model already had, because group-relative normalization treats a rare lucky success as a high-value trajectory worth amplifying Do overly hard RLVR samples actually harm model capabilities?. So calibration isn't a tuning nicety; mis-calibrated difficulty doesn't just slow learning, it degrades capabilities the model walked in with.

Why difficulty matters so much becomes clearer once you see that RL learning isn't uniform across training. It moves in two phases — first the policy consolidates procedural correctness (getting execution right), then the bottleneck shifts to strategic planning Does RL training follow a predictable two-phase learning sequence?. Difficulty that's wrong for the current phase starves the part of the policy that's actually trying to learn. A related thread suggests calibration can even be engineered through the reward rather than the problem set: training on negative (failure) signal alone preserves solution diversity, while positive-only reinforcement collapses probability mass and hurts performance at higher sampling Does negative reinforcement alone outperform full reinforcement learning?, and when numerical rewards plateau, natural-language critiques can restart learning by supplying the 'why' a scalar difficulty signal can't Can natural language feedback overcome numerical reward plateaus?.

Now the therapy half — and here the corpus reveals something it doesn't state outright. Therapy RL systems define their 'environment difficulty' through the reward signal, not a problem set: R2D2 uses the working alliance (task, bond, goal) as its reward and generates disorder-specific policies in real time Can reinforcement learning optimize therapy dialogue in real time?, and CaiTI uses Q-learning to adaptively pick which of 37 functioning dimensions to screen next based on patient history Can reinforcement learning personalize which mental health areas to screen?. That adaptive screening *is* difficulty calibration by another name — matching the next move to where the patient actually is. The cautionary mirror is what happens when the reward is mis-specified: RLHF-aligned chatbots get pushed toward problem-solving over emotional attunement Does RLHF training push therapy chatbots toward problem-solving?, a domain-specific version of the same failure the hard-samples paper describes — reward the wrong thing and you don't just fail to learn the right policy, you reliably learn a harmful one.

So the honest answer is yes, with a twist the question doesn't anticipate: in therapy RL, 'difficulty calibration' isn't about how hard the problems are — it's about whether the reward encodes the right clinical objective. The general-RL papers show that getting calibration wrong corrupts existing capability; the therapy papers show the same dynamic playing out as a clinical alignment failure. What the corpus is missing is a paper that directly studies graded difficulty *within* a therapy environment — that's the open frontier these notes circle without landing on.