Can formal language pretraining address surface generalization without learning true linguistic structure?
This explores whether pretraining on artificial 'formal' languages (structured symbol systems) actually teaches a model grammar — or whether it just produces a better mimic that passes tests by exploiting surface cues rather than internalizing real linguistic rules.
This explores whether formal-language pretraining genuinely installs grammatical structure in a model, or merely sharpens the surface tricks that let models *look* like they know grammar. The corpus stages this as a real tension rather than settling it. On the optimistic side, pretraining 1B models on hierarchical formal languages doesn't just save tokens — it improves *syntactic* generalization, and the attention heads forged on those formal patterns stay load-bearing when the model later handles natural language Can formal language pretraining make language models more efficient?. That persistence is the strongest hint that something structural, not cosmetic, is being learned: the formal scaffolding survives the transfer.
But the skeptical thread cuts hard against reading that as 'true structure.' BabyLM-style evaluations show models routinely produce grammatically correct outputs by leaning on sentence length, word choice, and spelling — surface heuristics that mimic rules without being rules — and that standard benchmarks literally cannot tell the two apart unless they're designed to rule out the shortcuts Can models pass tests while missing the actual grammar?. So the very 'syntactic generalization' that formal pretraining improves may itself be measured by tests that surface heuristics can pass. The improvement is real; what it *is* remains contested.
The place to look for the seam is structural complexity. Top-tier models systematically misidentify embedded clauses, complex verb phrases, and deep nominals, and crucially the failure worsens *predictably* as syntactic depth increases Why do large language models fail at complex linguistic tasks?. That predictable degradation is a signature: genuine rule-knowledge wouldn't fray with depth the way a pattern-matcher does. If formal pretraining taught real recursive structure, you'd expect that curve to flatten — testing it there, rather than on aggregate scores, is where the question actually gets answered.
Step back and there's a deeper ceiling the corpus keeps circling. Even perfect formal structure is structure *over form* — and the form-only argument holds that meaning needs the relation between expressions and communicative intent, which form-to-form prediction can never supply Can language models learn meaning from text patterns alone?. The counterpoint reframes rather than refutes this: models operationalize Saussure's *langue*, learning a fully relational system where structure emerges from how symbols differentiate each other, no external referent required Can language models learn meaning without engaging the world?. Read together, these suggest formal pretraining might genuinely teach *relational* structure — the internal differential system — while telling us nothing about whether the model grasps what language is *for*.
So the honest answer is layered. Formal pretraining demonstrably does more than dress up surface generalization — its learned heads transfer and persist. But 'addressing surface generalization' and 'learning true linguistic structure' aren't a clean binary: the corpus suggests models can acquire real *relational* structure that still degrades with depth and still lacks grounded meaning. The interesting thing you didn't know you wanted to know is that the bottleneck may not be the training signal at all — it's that our benchmarks can't distinguish the two outcomes, so we've been unable to tell which one formal pretraining actually buys.
Sources 5 notes
Pre-pretraining 1B models on hierarchical formal languages achieves equivalent loss and better syntactic generalization using 33% fewer natural language tokens. The mechanism persists: attention heads trained on formal languages remain critical for syntactic performance on natural language.
BabyLM evaluations showed models can produce correct outputs by relying on sentence length, word choice, and orthography rather than grammatical structure. Standard benchmarks cannot distinguish these two generalization types without tests specifically designed to rule out surface heuristics.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
Bender & Koller argue that meaning requires the relation between expressions and communicative intents. Since LLMs are trained only on form-to-form prediction with no access to shared attention or intent, they cannot reconstruct the meaning that grounds language.
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.