STaR trains AI by keeping only reasoning that got the right answer — but what grades it when there's no answer key?

This explores what replaces the answer key once STaR's simple self-teaching trick moves into domains where 'correct' can't be checked automatically. The original idea is very simple. A model writes out its reasoning, you keep only the attempts that reached the right answer, and you fine-tune on those. That alone pushed a small model on CommonsenseQA close to models 30 times its size Can models improve by filtering only on answer correctness?. The whole method depends on that filter, though. Take away ground truth and you need some other way to decide which self-generated reasoning deserves to be kept. The corpus has no single paper titled 'STaR without labels.' What it does have is several separate attempts to build that replacement filter, and each one shows a different way it can fail.

The closest descendant gives up on judging correctness and switches to a softer question. VeriFree doesn't ask 'is this answer right?' It asks 'given this reasoning, how likely is the model to produce the reference answer?' and uses that probability as both the reward and the training weight Can reasoning improvement work without answer verification?. It matches verifier-based methods on broad benchmarks like MMLU-Pro and GPQA. The catch is that it still needs a reference answer. What it removes is the need for an exact-match checker. So the first lesson is that 'no ground truth' usually means 'no cheap way to check,' not 'no target at all.'

The obvious next step is to let the model grade its own work, and this is where things go wrong. Models systematically over-trust answers they generated themselves, because a high-probability output simply feels right when they evaluate it Why do models trust their own generated answers?. More generally, LLMs are good at proposing candidates but poor at estimating how good those candidates actually are Can language models reliably judge their own candidate quality?. A STaR loop filtered by self-judgment can therefore reinforce whatever the model already believed. This matters because fluent, confident wrong answers already hide inside good-looking aggregate scores Why do confident wrong answers hide in standard accuracy metrics?. A self-training loop can quietly multiply exactly those errors.

The more promising replacements come from outside the training literature. One finding treats verification as its own scaling axis: judges get much more accurate when you break the criteria into parts, score more finely, and repeat the evaluation, all without retraining Can verification accuracy scale without training models?. Agent-style judges that actively gather evidence cut evaluation drift roughly 100-fold compared with a plain LLM judge Can agents evaluate AI outputs more reliably than language models?. The most unexpected parallel is in retrieval systems. One RAG design writes its own answers back into its knowledge base, but only after they pass entailment, source-attribution, and novelty checks Can RAG systems safely learn from their own generated answers?. That is the same pattern as STaR: the system generates, filters, and keeps what passes. The filter is just grounding in evidence instead of answer-matching.

What you might not have expected to learn: once ground truth is gone, the filter becomes the real thing being trained, and its errors compound with every round. Either it shrinks to a softer probability signal that still leans on references, or it has to become an evidence-checking system that is often more elaborate than the model it supervises.