Can models learn faster by predicting their own concepts?

The paper's claim is that latent representations a language model learns for itself can be made "first-class prediction targets" during pretraining rather than left as a byproduct of token supervision. NCP-ArchPreview adds Next Concept Prediction (NCP), which predicts "discrete concepts that span multiple tokens", alongside ordinary next-token prediction (NTP), while "preserving standard token-level autoregressive generation." At 8.9B parameters and 5.73T Dolma-3 tokens, the abstract reports that it reaches the final pretraining loss of OLMo-3-7B "by consuming only 51.3% of the total training tokens", then beats OLMo-3-7B by 2.45 points on the downstream macro-average, including 5.99 points on GSM8K. The authors describe this as "the largest demonstration of a latent-space language model to date."

The introduction gives the diagnosis. Models are known to form concept-like abstractions and latent world representations in their hidden states, but under NTP these arise "purely as an indirect byproduct" because supervision is "strictly confined to granular tokens", with no explicit objective on how semantic structure unfolds across multi-token spans. The remedy has three parts. A product-quantized concept vocabulary is built "directly from its hidden states", a dedicated Concept Module predicts future concepts, and the predicted concepts are fed back to the token level "to guide subsequent generation". NTP and NCP are trained jointly end to end. The discussion adds that controlled experiments show "complementary gains from the latent architecture, hierarchical routing, and NCP", and that scaling-law experiments indicate a 1.74× improvement in compute efficiency.

This sits between two neighbors. Can reasoning happen at the sentence level instead of tokens? also moves prediction above the token, but it drops tokens and works in a fixed external embedding space. Here the concept vocabulary comes from the model's own hidden states, is learned jointly, and sits beside a token-level objective that stays in place. The paper's result also runs in the same direction as Why is predicting latents more sample-efficient than tokens?, which proves a sample-efficiency advantage for latent prediction on a synthetic hierarchy. The 51.3% token figure is an LM-scale observation consistent with that theory, not a test of its mechanism. Because generation remains autoregressive, the result also does not bear on Does autoregressive generation uniquely enable LLM scaling?. It changes what is supervised, not the factorization.

The excerpt leaves a good deal open. It does not say what a concept corresponds to linguistically, how large the codebook is, or how the loss comparison with OLMo-3-7B was set up. It does not list the benchmarks in the macro-average, give variance across runs, or explain "hierarchical routing". It states the 85%-of-computation and 1.74× figures without their setup, and it does not test why the concept objective helps. It also does not say whether the fed-back concepts or the extra objective account for the gain. What the excerpt does support is narrower and still useful: at this scale, a learned discrete concept target added to NTP was trainable end to end, and the authors report lower loss per token and better downstream scores than a comparable open model.