Can optimal experimental design improve few-shot example selection?

In-context learning lets you put query-specific examples in the prompt, but which examples? AIPD frames this as active learning with optimal experimental design: starting from an unlabeled training pool, adaptively choose the examples whose labels would maximally reduce the LLM's prediction uncertainty across the test set, then pay to label only those. Two algorithms instantiate it — GO (minimize the variance of the posterior covariance for any test example) and SAL (simulate the impact of labeling each unlabeled example) — analyzed in linear models and shown to outperform other few-shot selection methods across small, medium, and large LMs.

The keeper is the reframing of demonstration selection as a budgeted experimental-design problem: under a labeling budget, the right examples are the ones that most reduce uncertainty over the actual test distribution, not the ones most superficially similar to the query. This is principled, test-set-aware example selection rather than heuristic retrieval.

This adds an active-learning lever to the vault's prompting/ICL thread. It complements Does learning from mistakes improve in-context learning? (LEAP — extract more from given examples) by addressing which examples to acquire in the first place, and the uncertainty-reduction objective echoes information-gain question-selection work elsewhere in the vault.