Can smarter system architecture replace hand-crafted AI reward functions — or do some problems need them no matter what?

This explores whether restructuring the *system* around a model — how tasks are decomposed, how rollouts are organized, how evaluation is gated — can substitute for hand-designed reward heuristics. The corpus says: surprisingly often, yes — but not universally, and the interesting part is *which* reward problems dissolve under better engineering versus which ones are irreducibly about the reward signal itself.

The strongest evidence for substitution comes from cases where structure manufactures a reward signal that used to be hand-built. Tree-GRPO uses the branching shape of its rollouts to turn a single end-of-trajectory reward into step-by-step process signals, by comparing sibling subtrees — eliminating the separate process reward model and step-level annotations people used to design by hand Can tree structure alone convert outcome rewards into process supervision?. MAKER goes further: by decomposing a task into minimal subtasks with per-step voting, it hits million-step reliability using small non-reasoning models, inverting the usual instinct to throw sophisticated reward shaping at hard problems Can extreme task decomposition enable reliable execution at million-step scale?. And the Darwin Gödel Machine swaps formal correctness proofs for plain empirical benchmarking, letting a system improve itself through trial and error rather than a designed objective Can AI systems improve themselves through trial and error?.

A second pattern is subtler: engineering doesn't replace the reward, it changes how the reward is *applied* so hand-tuning becomes unnecessary. DRO keeps rubrics but uses them as gates that accept or reject whole rollout groups rather than converting rubric scores into dense reward numbers — the structural move is what prevents reward hacking, not a cleverer reward formula Can rubrics and dense rewards work together without hacking?. AgentCompass makes the same bet on the evaluation side: decoupling benchmark, harness, and environment surfaces reward-hacking that scalar scores hide How can separating evaluation components make reward-hacking visible?. Even the harness-repository work shows that reorganizing *representation* — mapping runtime behavior to code — lets a weak planner match a strong one, a capability gain bought with engineering rather than reward design Can explicit behavior maps help weaker planners compete with stronger models?.

But the corpus also draws a hard line. Some failures are baked into the reward's mathematics and no amount of plumbing fixes them: binary correctness rewards *provably* wreck calibration by rewarding confident wrong guesses, and the fix is a reward-level one — adding a Brier scoring term — not a system rearrangement Does binary reward training hurt model calibration?. Where the reward target is genuinely subjective, you still need to *design* what counts as good, though checklist decomposition converts that design into verifiable sub-criteria rather than a hand-tuned holistic score Can breaking down instructions into checklists improve AI reward signals?. And reward models themselves get better not from engineering around them but from letting them reason before scoring Can reward models benefit from reasoning before scoring?.

The reframe worth leaving with: several problems that *look* like they need a hand-designed reward heuristic turn out to be measurement artifacts that better instrumentation exposes as illusory. The exploration–exploitation trade-off — long treated as a fundamental tension to be balanced by reward tuning — shows near-zero correlation once you measure hidden states instead of token-level statistics; it was an artifact of *how* you looked, not a real trade-off to engineer around Is the exploration-exploitation trade-off actually fundamental?. Even the mechanism of RL is more structural than we design for: updates concentrate in a sparse, nearly identical subnetwork across seeds without anyone regularizing for it Does reinforcement learning update only a small fraction of parameters?. So the honest answer is a split: system-level engineering can replace most reward heuristics that were really compensating for bad structure or bad measurement — but it cannot replace the ones encoding what you actually want the model to value.