What makes a good in-context learning example for a given task?
This reads the question as 'what actually makes a demonstration useful' — not just picking examples that look similar to the task, but what property of an example does the teaching work.
This explores what makes an in-context example earn its place in the prompt — and the corpus pushes back hard on the obvious answer, which is 'pick examples that resemble the test case.' Several lines suggest similarity is a weak proxy for usefulness, and that the better question is what an example *demonstrates*. The most counterintuitive finding: an example's main job may be teaching the *shape of the answer* rather than the content. Models trained on semantically empty or even deliberately wrong instructions perform almost as well as those given correct ones — what transfers is knowledge of the output space, not task understanding Does instruction tuning teach task understanding or output format?. By that logic, a good example is one that vividly shows the model what a valid answer looks like, format and all.
If you do want to select examples deliberately, the corpus argues against grabbing the nearest neighbors. Framing demonstration choice as a budgeted experiment — pick the examples that most reduce uncertainty across the whole test set — beats similarity-based retrieval across small, medium, and large models Can optimal experimental design improve few-shot example selection?. The good example isn't the one closest to your query; it's the one that resolves the most ambiguity about the task as a whole. And sometimes a single, well-chosen example is enough to flip latent capability on: one training instance lifted math accuracy from 36% to 73.6%, suggesting examples can act as activation signals rather than lessons Can a single training example unlock mathematical reasoning?.
The richest thread reframes the unit of a 'good example' entirely. A clean correct demonstration may teach less than one that surfaces the *principle* behind the answer. LEAP shows models improve by being induced to err on the few-shot examples, then reflecting on those mistakes to derive explicit task rules — error-revealing examples beat tidy ones Does learning from mistakes improve in-context learning?. Relatedly, extracting natural-language skills from context — turning examples into reusable rules — lifts frozen models without any weight updates Can frozen models learn better by extracting context into skills?. For sequential or multi-step tasks, the unit isn't even a single example: in-context learning of decision-making requires full or partial *trajectories* from the same setting, not isolated input-output pairs Why do trajectories matter more than individual examples for in-context learning?.
Two hard ceilings bound all of this. No example can inject knowledge the model never had — prompting only reorganizes and activates what's already in the training distribution Can prompt optimization teach models knowledge they lack?. And even a well-chosen example can be ignored: when a model's prior training associations are strong enough, in-context information loses, and textual prompting alone can't override it Why do language models ignore information in their context?. So the honest answer is layered: a good example shows the output shape, reduces task-wide uncertainty rather than just matching the query, ideally exposes a principle (often through a corrected mistake), and supplies a full trajectory when the task is sequential — but none of it works against a knowledge gap or an overpowering prior.
Sources 8 notes
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
AIPD frames demonstration selection as budgeted active learning, choosing examples that maximally reduce test-set uncertainty. Two algorithms (GO and SAL) outperformed similarity-based methods across small, medium, and large language models.
A single example in RLVR boosts math performance from 36% to 73.6% and enables test accuracy to improve for 1,400 steps after training accuracy reaches 100%, revealing that minimal activation signals unlock latent reasoning capability.
LEAP demonstrates that models achieve better performance on reasoning and math tasks by intentionally erring on few-shot examples, reflecting on mistakes, and deriving explicit task-specific principles—without additional labeled data or fine-tuning.
Extracting natural-language rules from context into reusable skills improves frozen model reasoning without weight updates. On CL-bench, this lifts GPT-4.1 from 11.1% to 16.5%, with skills transferable across model backbones.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.