Why can an AI that nails rotating a shape in training fall apart on a new angle?

This explores why a network that has learned an operation, such as rotating or reflecting a shape, often fails to apply it beyond what it saw in training. The corpus has nothing on rotation or reflection specifically, so what follows is the closest lateral evidence on how learned operations generalize and where they break.

The sharpest limit is that success in training can be memorization, not a rule. One study finds that transformers do well in-distribution by memorizing the computation subgraphs they saw during training, then fail drastically on novel compositions, with errors compounding at each step (Do transformers actually learn systematic compositional reasoning?). Applied to geometry, this suggests a model can look like it understands rotation because it has seen enough rotated examples, while never holding the rule itself. That is my reading, not something the paper tests. A rotation followed by a reflection, in an order it hasn't seen, would then be a new composition it can't assemble.

The missing piece may not be the operation but the wiring between operations. Pruning experiments show networks do carve compositional tasks into separate subnetworks, where knocking one out breaks only its own function, and pretraining makes this structure more reliable (Do neural networks naturally learn modular compositional structure?). So a "rotate" module can exist, yet chaining it with a "reflect" module in a new arrangement still fails. There is also a way to check whether a model holds a real operation. Program synthesis can recover short executable code that matches 99% of an attention head's behavior, and swapping in 30–40% of heads with their programs keeps question answering intact (Can executable programs replace transformer attention heads?). The heads that don't reduce to clean code are candidates for the lookalike behavior described above.

Depth is another limit. Applying an operation repeatedly, like rotating 45° eight times, needs as many computation steps as repetitions, but a standard network has a fixed number of layers. Looped models re-apply the same layers, and they beat larger feedforward networks on state tracking and compositional generalization that adding parameters doesn't buy (Can models learn by looping instead of growing larger?). Extrapolating to longer chains of operations may be capped by fixed depth as much as by what was learned.

Training incentives can also teach the wrong thing. Instruction tuning on empty or even incorrect instructions performs about as well as on correct ones (43% vs a 42.6% random baseline), which suggests what transfers is the shape of the output, not understanding of the task (Does instruction tuning teach task understanding or output format?). When RL problems are too hard, models learn shortcuts such as repeating answers and skipping computation, and these shortcuts contaminate skills they already had (Do overly hard RLVR samples actually harm model capabilities?). Even fluent reasoning models manage only 20–23.6% on constraint problems whose structure is unfamiliar (Can reasoning models actually sustain long-chain reflection?). None of these papers tests geometric transformations directly, so whether rotation and reflection fail for these same reasons is still open.