Some AI-agent tricks work across models and tasks, while others only work on the test they were built for. What's the difference?

This explores what separates an agent trick that keeps working when the model, task, or benchmark changes from one that only works on the setup it was built for. The collection has no single study that tests this head-on. Its evidence points to a consistent answer: reusable mechanisms act on the shape of the work, and benchmark-specific ones act on the content of particular tasks. When one team ran an automated search for harness improvements across many environments instead of one, it found four mechanisms: how actions execute, how context gets compacted, how observations are handled, and how reading gets delegated. None of them mention any particular task, and together they cut token traffic by 44.7–49.0% (Can agent harnesses be automatically optimized across many environments?).

The second marker is abstraction. Agent Workflow Memory learns routines at the level of sub-tasks, smaller than whole tasks, and strips out example-specific values such as the particular product or date. It then stacks those routines into larger ones. Its results give a usable test for reusability: the gains grew as the gap between training and test tasks widened (Can agents learn reusable sub-task routines from past experience?). A benchmark-specific trick should get worse as the test looks less like the training data, and a reusable one shouldn't.

The third marker is where the mechanism lives. Reliable agents move memory, skills, and interaction protocols out of the model and into the harness, so the model doesn't have to re-solve the same problems each time (Where does agent reliability actually come from?). That layer is also the easiest to change and undo. Recent self-improvement work has concentrated on updating prompts, memory, and tools, not weights (Do self-improving agents really split into two distinct loops?). The sharpest evidence is a single runbook that lifted several frozen models on Terminal-Bench 2.1, reaching 95.3% on GPT-5.6 and adding 5.4 points to DeepSeek-V4 Flash, and that carried over to newer models without modification (Can execution harnesses lift model performance without retuning weights?). A mechanism fitted to one model's quirks wouldn't survive a model swap. One fitted to the structure of the problem does.

You also need to check that a mechanism is doing the work you think it is. One finding is that about 80% of multi-agent performance variance tracks token budget, not coordination cleverness (How does test-time scaling work at the agent level?). A coordination scheme that looks brilliant may just be spending more. Identical success rates can also hide huge differences in efficiency and reliability, which is why evaluators are moving toward trajectory-level measures (How should we measure agent system performance beyond task success?). For memory, module-by-module scoring shows which stage (storage, extraction, retrieval, or maintenance) earned the result (How should we actually evaluate agent memory systems?).

Together these give a practical checklist. A mechanism is probably reusable if it survives a model swap, keeps helping as tasks drift from the training examples, and holds up when the token budget is held fixed. It should also show its effect in the trajectory, not only in the final score.