Does a longer list of labeling rules make AI worse at sorting interview or survey text — and does similar data help?

This explores whether the length of a coding scheme and the similarity of the data change how accurately an LLM labels text. The corpus has no note that varies either one, and only one note is about LLM coders at all. What follows is what that note offers, plus a few adjacent findings that suggest hypotheses. Those hypotheses are extrapolations, not results.

The LLM-coding note is Does disagreement between AI coders signal better accuracy?. In multi-agent coding setups, accuracy was higher when the agents argued at length and left some labels undecidable. So disagreement is a usable diagnostic. If you're testing whether a codebook is too long or its categories overlap too much, watch where the coders split. The note says that split is a signal about interpretive difficulty, not automatically a failure. It doesn't say how codebook size changes the amount of disagreement.

On data similarity, the nearest evidence comes from optimization tasks, not coding. Models tend to recognize a problem as resembling something familiar and emit a plausible answer (Do large language models actually perform iterative optimization?). When the test set is a variation on the training problems, performance drops sharply (Do fine-tuned language models actually learn optimization procedures?). A parallel framing says tasks whose correct answer is low-probability text are systematically harder (Can we predict where language models will fail?). If LLM coding works the same way, data that looks like common, well-represented text should be coded more accurately, and unusual data should be coded less accurately. That is a prediction to test, not something these notes show for coding.

On codebook length, two notes hint at mechanisms. A long codebook means many rules applied at once, and LLMs plateau at about 55–60% constraint satisfaction on constrained-optimization tasks regardless of model scale (Do larger language models solve constrained optimization better?). That points to a ceiling on how many rules a model reliably applies at once, not a gap that scaling will close. The other is a design idea from a long-document reader: it keeps compressed gist memories and looks up details only when needed (Can LLMs read long documents like humans do?). By analogy, a very long codebook might work better if the model sees a summary first and pulls in only the relevant category definitions per passage. The note itself is about reading documents, not codebooks.

The practical gap is measurement. Errors from stronger models can look fine on the surface, so eyeballing outputs won't reveal accuracy loss (Does model capability change how documents degrade?). To answer your question with data, you would run the same texts under short and long codebooks, split the texts into familiar and unusual groups, and compare against human labels. Disagreement between coders would give you a cheap early warning along the way.