How much does the user shape what a model generates?

The Foundation Priors paper models prompt engineering not as instruction-giving but as an iterative alignment process. The user proposes a query to the foundation model, evaluates the resulting synthetic data against their anticipated distribution (using a divergence measure), and refines the prompt until the synthetic data aligns sufficiently with those priors. The end product captures both the foundation model's learned patterns and the user's subjective filters.

This reframes what prompt engineering actually does. The standard view treats it as unlocking model capability — finding the right key for the right lock. The Foundation Priors view treats it as imposing user subjectivity onto model output — the user is not discovering what the model knows but shaping what the model produces to match what the user already expects. The "skill" of prompt engineering is partly the skill of iteratively refining until output confirms prior expectations.

The epistemic danger is clear: without external anchoring in real data, this process produces epistemic circularity. The user refines prompts until the output looks right, where "looks right" means "matches what I already believe." The model becomes a mirror that reflects the user's anticipated distribution back to them with the authority of a computational system. Since How should users control systems with unpredictable outputs?, the unpredictability of generation creates an illusion of independent inquiry — the output varies enough to feel like discovery rather than confirmation, but the prompt refinement process systematically steers toward the user's priors.