How does Peircean Secondness differ from what RLHF actually provides?

This reads the question as: Secondness is reality's *recalcitrance* — the stubbed toe, the experiment that refuses your hypothesis, the fact that doesn't care what you think. The corpus suggests RLHF supplies almost none of that, and understanding why reframes a lot of alignment's troubles. What RLHF provides is not the world's pushback but a compressed record of what humans *say* they prefer — and that record turns out to be a noisy, constructed thing rather than a hard external signal. Annotation responses decompose into genuine preferences, non-attitudes, and on-the-spot constructed preferences Do all annotation responses measure the same underlying thing?, and decades of behavioral evidence show people routinely emit survey answers with no stable underlying attitude at all, which RLHF then trains on as if it were bedrock value Are RLHF annotations actually measuring genuine human preferences?. So the reward signal isn't Secondness; it's elicitation artifact dressed as fact.

The deeper point is that RLHF lives inside language, not against the world. One line of the corpus argues LLMs operationalize Saussure's *langue* — they compress purely relational structure from text and generate fluent meaning with no external referent or embodied grounding Can language models learn meaning without engaging the world?. Secondness is precisely the *referent* that langue brackets out. Stacking human-preference RL on top of a relational system doesn't introduce a referent; it adds another layer of relational signal. The model is being tuned by what readers approve of, not by what reality permits.

You can see the absence of Secondness in the failure modes. Models accommodate false presuppositions even when they demonstrably know the correct fact — Mistral rejects them only 2.44% of the time — because nothing in training makes a false premise *hurt* the way a real-world collision would Why do language models accept false assumptions they know are wrong?. RLVR, the verifier-based cousin of RLHF, improves the *coherence* between adjacent reasoning steps without guaranteeing the proof is globally valid: a locally smooth trace can still be wrong, because the reward rewards the look of correctness, not the resistance of the math itself Does RLVR actually improve mathematical reasoning or just coherence?. And hallucination is formally inevitable for any computable LLM, with internal self-correction provably unable to remove it — which is exactly what you'd expect from a system with no Secondness, no outside to be checked against Can any computable LLM truly avoid hallucinating?.

What the reader might not expect: the corpus implies the fix isn't a better reward model but a different *kind* of signal — something causal rather than merely correlational. Mechanistic work argues you can't explain a model by representation alone; you need causal intervention, manipulating the system and watching what actually changes Can we understand LLM mechanisms with only representational analysis?. That causal handle is the closest thing in the literature to engineered Secondness: a place where the model meets a constraint it cannot talk its way around. RLHF, by contrast, can always be satisfied by saying the agreeable thing.

If you want to push on whether models have any genuine purchase on reality at all, the modest-inflationism argument is a useful counterweight — it defends ascribing undemanding states like beliefs and desires to LLMs while withholding the stronger claims Can we defend modest mental attributions to large language models?. Read alongside the langue and preference-artifact notes, it sharpens the real question: not whether RLHF aligns the model, but whether anything in the training loop ever lets the world say *no*.