Can attribute-specific preference optimization improve question quality in information-seeking?
This explores whether breaking 'question quality' into named sub-attributes — and optimizing each one separately — actually produces better questions when a system is trying to gather information, rather than training against a single lumped quality score.
This explores whether breaking 'question quality' into named sub-attributes — and optimizing each one separately — produces better questions in information-seeking, versus training against one lumped quality score. The corpus has a direct, encouraging answer and several flanking results that explain *why* it works.
The centerpiece is the ALFA framework Can models learn to ask genuinely useful clarifying questions?, which decomposes question quality into theory-grounded attributes like clarity, relevance, and specificity, then trains on 80K attribute-specific preference pairs. The headline result: attribute-specific optimization beats single-score training, and the gap is widest in clinical reasoning, where asking the *right* clarifying question changes the downstream decision. So the answer to your question is yes — but the interesting part is the mechanism. A single quality score is a blurry target; a good question and a mediocre one can land on similar scalars for different reasons. Factoring quality into attributes gives the optimizer cleaner, less-entangled gradients to follow.
That 'factor the signal so gradients stop interfering' idea is not unique to questions — the corpus keeps rediscovering it. Conversational recommenders find that fusing what-to-ask, what-to-recommend, and when into one policy beats optimizing them in isolation, precisely because separated components can't pass gradient signal to each other Can unified policy learning improve conversational recommender systems?. Personalization research shows the same shape from the reward side: factorizing a user's preferences into a linear combination of base reward functions lets ten adaptive questions pin down someone's coefficients Can user preferences be learned from just ten questions?. The recurring lesson is that the *granularity* of your training signal matters as much as its volume.
There's also a prior question worth surfacing: can a model learn to ask at all, rather than guess? It can — reinforcement learning lifted proactive 'I'm missing information, let me ask' behavior from near-zero to ~74% on deliberately under-specified problems, and notably the skill is fragile without explicit training Can models learn to ask clarifying questions instead of guessing?. Attribute optimization is the natural next layer on top: first teach the model *to* ask, then teach it to ask *well* along specific dimensions. And whether attributes are even the right axes depends on the task — non-factoid work shows different question types (comparison, debate, experience) demand genuinely different handling, so 'quality' isn't one thing across the board Does question type determine the right retrieval strategy?.
One caution the corpus hands you for free: be careful what you measure. Supervised fine-tuning can raise final-answer accuracy while quietly degrading the quality of the reasoning steps that produced it, because standard metrics only score the endpoint Does supervised fine-tuning improve reasoning or just answers?. The deeper argument for attribute-specific optimization is that it refuses that trap — by scoring clarity, specificity, and relevance directly, it measures the qualities you actually care about instead of trusting a single number to stand in for all of them.
Sources 6 notes
The ALFA framework breaks down question quality into theory-grounded attributes (clarity, relevance, specificity) and trains models on 80K attribute-specific preference pairs. Attribute-specific optimization outperforms single-score training, especially in clinical reasoning where asking the right clarifying question directly impacts decision quality.
Research shows that formulating attribute-asking, item-recommending, and timing decisions as a single graph-based RL policy achieves better joint optimization than isolated components. Separation prevents gradient signals from informing one another and fails to optimize conversation trajectory holistically.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
Reinforcement learning training increased proactive critical thinking accuracy from 0.15% to 73.98% on deliberately flawed math problems. Notably, inference-time scaling degraded this ability in untrained models but improved it after RL training, suggesting the capability is learnable but fragile without explicit training.
Research shows non-factoid questions split into five types, each requiring different retrieval and aggregation methods. Evidence-based questions suit standard RAG, while debate and comparison need aspect-specific retrieval, and experience/reason questions need decomposition or filtering strategies.
Supervised fine-tuning improves final-answer accuracy on benchmarks but cuts Information Gain by 38.9 percent, meaning models generate correct answers through post-hoc rationalization rather than genuine inferential steps. Standard metrics miss this degradation because they only measure final correctness.