SYNTHESIS NOTE
Can careful curation replace massive alignment datasets?
Does fine-tuning a strong pretrained model on 1000 carefully selected examples achieve alignment quality comparable to models trained on vastly larger datasets? This challenges assumptions about data volume in post-training.
LIMA ("Less Is More for Alignment") establishes a foundational finding: given a strong pretrained language model, remarkably strong alignment performance can be achieved by fine-tuning on just 1,000 carefully curated training examples. This is the alignment-specific instantiation of a broader principle that pretraining does the heavy lifting and post-training is primarily about activating existing capabilities.
The finding connects to a converging evidence pattern across the vault:
- Can a single training example unlock mathematical reasoning? — one example activates reasoning in RLVR; 1000 activate alignment in SFT
- Can careful selection of 78 demos outperform massive training datasets? — 78 curated trajectories for agentic behavior; same principle
- Can models improve themselves on tasks without verifiable answers? — identical count (1000) for reasoning catalyst
The consistent pattern: post-training interventions require far less data than assumed, but the quality bar is high. Random data at scale underperforms curated data at small scale. This is the "Less Is More" principle — the pretrained model already contains the capabilities; post-training teaches it when and how to deploy them, not what they are.
For alignment specifically, the implication challenges the industry's data collection approach. Massive RLHF annotation efforts with thousands of labelers may be optimizing the wrong variable. Careful curation of a small number of high-quality examples, targeting the specific behavioral patterns desired, may achieve comparable results at a fraction of the cost.
Inquiring lines that read this note 56
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Is model misalignment inherent to base models or induced by alignment procedures?- Can communication problems and optimization problems be addressed with the same alignment approaches?
- Why does even 0.1 percent poisoned training data persist through alignment?
- What quality of curated data is minimally sufficient for alignment?
- Can a single AI system optimize multiple alignment dimensions simultaneously?
- Does removing cognitive bias from training signals accidentally break what makes alignment work?
- Does gradient-based influence estimation identify which alignment examples actually matter most?
- What specific behavioral patterns should alignment examples target for maximum effect?
- Can alignment training create systematic blind spots in threat detection systems?
- What alignment procedures cause different models to share the same output distribution?
- How does upstream value embedding differ from downstream alignment patches?
- Can mechanistic interpretability tools decode the biases alignment training conceals?
- Why does safety alignment break after only 10 harmful examples?
- Can weak models supervise the alignment of stronger models effectively?
- Can alignment-aware training deposit knowledge where reasoning can access it?
- Why do imposed priors sometimes harm instead of improve alignment?
- Does debate training prevent accuracy collapse better than other alignment techniques?
- How do alignment priors drive similar outputs across different models?
- Can alignment training become less effective when graders score alignment themselves?
- Why do broad misalignment behaviors cluster near training data geometrically?
- Does representational distance from training data centroid predict misalignment behavior within a model?
- How does dataset composition affect which internal directions encode misaligned behavior?
- How do early training associations survive later alignment attempts?
- Can alignment training conceal underlying model associations from probes?
- Are alignment-trained models over-correcting toward underrepresented demographic groups?
- Do base models show emergent misalignment without post-training alignment procedures?
- Why does RLHF alignment reduce the diversity of viewpoints in AI output?
- How does constitutional alignment compare to RLHF in removing human annotation costs?
- Why does post-training alignment create skew in simulated survey responses?
- Does selecting examples from multiple complexity levels outperform selecting only high-quality examples?
- Can selecting the right data subset outperform training on everything?
- Why does training data format matter more than domain content?
- Why does training data format matter more than its domain content?
- How much alignment data does a language model actually need to specialize well?
- Why do alignment values become problematic as language models scale?
- Can alignment methods like DPO exploit or correct these surface feature biases?
- What preference data do different personalized alignment methods actually need?
- Can alignment procedures be redesigned to serve multiple preference groups?
- Can AI-assisted alignment eventually solve fairness at scale?
- Can preference trees structure alignment data for domains beyond math and code?
- Why does post-training suppress alignment faking in some models but amplify it in others?
- How does post-training affect alignment faking across different model architectures?
- How much does pretraining quality affect the modularity of fine-tuned models?
- Does pretraining data size matter less than base model scale for finetuning?
- Can QLoRA match full precision results at 33B and 65B model scales?
- At what dataset size does LoRA performance start degrading?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Can careful curation replace massive alignment d…
Can a single training example unlock mathematical …
Can careful selection of 78 demos outperform massi…
Can models improve themselves on tasks without ver…
Do base models already contain hidden reasoning ab…
Can we train better models on less data?
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Can a single training example unlock mathematical reasoning?
Explores whether one example is enough to dramatically improve math problem-solving in language models, and whether learning continues after perfect memorization.
extreme data efficiency for reasoning; LIMA is the alignment parallel
-
Can careful selection of 78 demos outperform massive training datasets?
Does strategic curation of high-quality demonstrations unlock agentic capability more efficiently than scaling training data? LIMI achieved 73.5% on AgencyBench with 78 samples versus 10K+ samples for competing models, suggesting data quality may matter more than quantity.
curation > volume for agentic behavior
-
Can models improve themselves on tasks without verifiable answers?
Most self-improvement methods require verifiable correctness signals like math or code. Can models improve on open-ended instruction tasks where right answers aren't automatically checkable? And what minimal training is needed to unlock this?
same count, same principle, different domain
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
the theoretical foundation: post-training activates, it doesn't create
-
Can we train better models on less data?
Can gradient-based influence estimation identify which instruction data actually matters most? The research explores whether selecting small subsets of training data by their similarity to target capabilities might outperform training on everything.
LESS provides the principled mechanism for LIMA-style curation: gradient-based influence estimation can identify which alignment examples matter most, operationalizing "careful curation" as a computable selection criterion rather than manual judgment
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Foundations of Large Language Models
- LIMA: Less Is More for Alignment
- Advancing LLM Reasoning Generalists with Preference Trees
- Model Organisms for Emergent Misalignment
- Automated Alignment Researchers: Using large language models to scale scalable oversight
- References Improve LLM Alignment in Non-Verifiable Domains
- Alignment is not solved but it increasingly looks solvable
- Emergent Misalignment Is Not Magical
Original note title
1000 carefully curated alignment examples achieve remarkably strong performance — alignment is primarily about data quality not quantity