Teaching an AI to predict ideas alongside words seems to speed up training — but does the size of its idea vocabulary matter?

This explores whether the number of distinct concepts a model predicts (its concept vocabulary) changes the efficiency payoff of joint training. The corpus doesn't answer that directly. The one note on joint training reports a single headline result and doesn't say how it varies with vocabulary size. An 8.9B model trained to predict both next tokens and learned concepts from its own hidden states matched OLMo-3-7B's final loss using only 51.3% of the training tokens, and beat it by 2.45 points downstream Can models learn faster by predicting their own concepts?. Nothing in the retrieved material compares small and large concept vocabularies, so whether the gain grows, plateaus or reverses as the vocabulary changes is an open question here.

Several neighboring notes suggest what to look for. The nearest is about how the supervision is organized. StructTuning reached 50% of full-corpus performance with 0.3% of the training data by arranging text into an auto-generated domain taxonomy, so the model learns where a fact sits in a conceptual structure instead of raw text patterns Can organizing knowledge structures beat raw training data volume?. If concept prediction works the same way, the useful thing is probably the structure the concepts impose, and the raw count of concepts may matter less. That is a guess. The notes don't test it.

The second angle is granularity, and it shows a possible cost of coarse concepts. WordNet analysis finds general concepts (hypernyms) outnumber and outoccur specific ones, and LLMs favor frequent wording, so output drifts toward abstraction and loses expert-level specificity Does word frequency correlate with semantic abstraction?. A concept vocabulary built from what is common would carry the same skew. A small vocabulary of broad concepts might speed up learning of general structure while giving little signal about specialist distinctions. Again, that's an inference from the frequency finding, not something the joint-training note measured.

The third angle is whether concept count is even the right scaling axis. Latent-Thought Language Models scale few-shot reasoning along two axes, model size and latent size, by coupling fast local learning of the latent vectors with slow global learning of the decoder Can latent thought vectors scale language models beyond parameters?. That result treats the size of the intermediate representation as something you can scale independently of parameters. It's the closest thing here to a concept-size dial, though it covers continuous latents rather than a discrete vocabulary. Separately, RLVR work finds that only about 20% of tokens carry the learning signal, and training on just those matches full updates Do high-entropy tokens drive reasoning model improvements?. That hints that gains from any auxiliary signal depend on how well it lands on the few decision points that matter.

The corpus establishes that joint concept-and-token prediction can roughly halve the tokens needed. It has no data on how concept vocabulary size modulates that. The best test would vary the vocabulary from coarse to fine and measure token savings at each level, and the notes above suggest the answer may depend more on the concepts' structure and granularity than on their count.