Why did teaching a computer to recognize a simple picture once take so much raw computing muscle?

This explores why neural networks have historically needed so much computing power even for tasks that seem simple, like recognizing images. To be upfront: this collection doesn't cover early image networks, GPUs, or the history of computer vision. Those papers aren't in the corpus. What it does have is a lot of research on a closely related question: where a network's computing power actually goes, and how much of it is wasted. Read that way, the answer is surprising. Much of the expense was never required by the task itself. It came from the way standard networks are built.

The basic issue is that a standard network learns by adjusting every connection, and it does this with very general building blocks. Can learnable spline activations beat fixed MLP designs? makes this concrete. Classic networks use fixed, simple activation functions and make up for that simplicity with a huge number of connections. When the connections themselves can learn more flexible shapes, smaller models match or beat bigger ones. A related finding is Does depth matter more than width for tiny language models?. It shows that how you arrange the parameters matters too: stacking more thin layers gets more out of the same budget than spreading the parameters across fewer, wider layers. Both papers suggest that brute-force size has partly been a stand-in for better design.

A second clue is how networks use the capacity they already have. Is representational sparsity learned or intrinsic to neural networks? finds that networks only build rich, dense internal activity for inputs they've seen often. Unfamiliar inputs get sparse, thin representations. So a network needs a lot of exposure, which means a lot of training compute, before it handles even a 'simple' category well. Can sparse weight training make neural networks interpretable by design? approaches the same idea from the other side. When most connections are forced to zero, a task can often be handled by a small, clean circuit. That implies dense networks carry far more machinery than any single task needs. Do neural networks naturally learn modular compositional structure? backs this up: networks tend to solve each subtask in its own small subnetwork, while the rest of the model sits idle for that job.

The idea you might not have expected: a network can spend all that compute and still learn the wrong thing internally. Can AI pass every test while understanding nothing? argues that networks trained the standard way can give perfect answers while their internal organization is tangled and incoherent. Standard tests can't detect this. So the cost of a 'simple' task can include learning a messy solution that only looks clean from the outside.

The collection's answer to 'does it have to be this expensive?' is mostly no. Recent work squeezes out the waste in several ways. Can we finetune huge language models on a single GPU? shrinks the memory needed for fine-tuning by more than 15 times. Can models learn by looping instead of growing larger? gets more reasoning out of a model by reusing the same layers several times instead of adding new ones. Can spiking neurons make transformers efficient on any hardware? borrows ideas from how brain cells fire to run models more cheaply on a wider range of hardware. These papers are about today's language models, not early image networks, but they treat the same old problem as a design flaw that can be fixed. They don't treat it as a fixed price of intelligence.