Can agents learn from vague goals without predefined metrics?

Aspire starts from a contrast the introduction states directly. Human learning often begins with "a broad capability direction rather than a predefined benchmark, training set, or fully computable reward function," and so involves three coupled decisions: what to improve, how to improve it, and how to verify the improvement. Existing LLM self-evolution work "focuses primarily on the second decision," searching for effective optimization paths toward a specified objective, which the abstract describes as "reducing self-evolution to optimizing an explicit objective rather than deciding what and how to learn."

The benchmark removes the scaffolding that makes the second decision tractable. The agent receives only a natural-language capability goal, and the downstream evaluation tasks stay hidden. It must operationalize the goal by choosing data and update methods, constructing its own training and validation signals, and deciding when to evaluate. The same interactive environment supports both model-weight evolution and agent-harness evolution, and the outcome is scored on a hidden, expert-authored set of 520 items across six goals. The stated finding is that "vague goals redirect search effort toward goal interpretation," and the conclusion adds that the process slice associates vague-goal prompting with more goal-definition work.

The discussion turns this into a caution about measurement. Self-directed runs "produce evaluated checkpoints far more often than retained improvements," and the highest successor-harness mean stays below a fixed reference under unchanged runtime weights. The authors conclude that progress has to be judged against the base model rather than only against the previous checkpoint, because "closing the training loop is not yet the same as closing the capability loop."

Against the nearest notes, this paper moves the open problem one step earlier than they do. Can agents evolve their own objectives during search? has SAGA's outer loop propose objectives and compile them into executable scoring functions; Aspire is a test of the harder case where no scoring function exists yet and evaluation stays out of the agent's sight. Can AI systems improve themselves through trial and error? improves against fixed benchmarks, which is exactly the setting the introduction calls the existing norm. The weights-versus-harness support echoes Do self-improving agents really split into two distinct loops?. The third decision, verification, is where What limits how much models can improve themselves? would apply, though the excerpt does not itself make that link.

The excerpt is silent on the numbers behind these results: the size of the gap to the fixed reference, which models and harnesses were run, and what the reference is. The authors bound their own conclusions by the six goals, the coverage and scoring quality of the expert-authored items, one canonical adaptive-feedback run per configuration and goal, and trace content available only under controlled access. What it supports is narrow: in this benchmark, an evaluated checkpoint is a weak proxy for retained capability, so any claim of self-evolution from a vague goal needs a base-model comparison, not only a comparison with the previous checkpoint.