Do long-horizon benchmarks actually measure decision quality?

The paper's central claim is that existing long-horizon benchmarks measure the wrong unit. They "only report whether the agent finishes the task and provide no measure of the quality of the decisions made along the way." The authors call the ability to make those decisions well the "taste" of an agent, and say that "none of" the existing benchmarks measures it. Their instrument is Taste-Bench, and on it "the best model answers only 59.7% of the questions correctly." The decisions in question are of the kind "which hypothesis to test, which implementation to build on, or which experiment to run next."

The mechanism the introduction gives is a delay between decision and consequence: "a wrong decision often looks reasonable at the moment, and its cost appears only much later, after the agent has spent a large part of its budget." An end-to-end score collapses that delay into one pass or fail at the end of the run, so it cannot say which choice caused the outcome. Taste-Bench isolates the choice instead. Each question is a decision fork, "a point in a trajectory where multiple directions are available and one of them leads to a better outcome," and the model picks a direction "without seeing what happens after the fork." Forks are mined automatically from parallel attempts at the same task and from detours inside a single trajectory, from trajectories that agents produced on engineering and research tasks, "without needing human annotation."

Against the neighbors, this adds a decision-quality measure inside the long run. Does a single benchmark score actually predict agent readiness? argues that success-only scoring hides other axes, and taste is a candidate for one such axis. It sits within the trajectory, not in the deployment context that note's examples mostly come from. What predicts success in ultra-long-horizon agent tasks? also looks inside the run, but at whether the agent keeps cycling through feedback. Taste asks about the direction chosen at a branch point, which persistence alone would not isolate. Do automated benchmarks hide what frontier AI systems can really do? reaches into long tasks through small-sample qualitative log analysis. Taste-Bench mines logged trajectories automatically, a different route to the same material.

The excerpt is only the abstract and an introduction passage, and it leaves much open. It does not give the number of questions, the models tested beyond the best one, or how many directions each fork offers, so it cannot say how far 59.7% sits above chance. It does not say how "a better outcome" is determined for a fork, or whether taste scores predict end-to-end success, which is the test of whether taste is separable from what benchmarks already measure. The "improving" half of the title has no method in the excerpt. What the passages do support is narrower: a stated gap in what benchmarks report, and a fork-based way to probe it that needs no human labeling.