Why does masking future experts guarantee causal validity without external verification?
This explores why TiMoE's trick of blocking experts trained on later time periods makes its answers causally honest by construction — rather than needing a separate check after the fact to confirm no future knowledge leaked in.
This explores why masking future-dated experts is a *structural* guarantee rather than a verified one. The TiMoE design pre-trains separate experts on disjoint two-year slices of data, then refuses to route a query to any expert whose window postdates it Can routing mask future experts to prevent knowledge leakage?. The reason no external verification is needed is simple but easy to miss: the model can't leak future knowledge because the future knowledge is never *reachable*. You don't audit outputs for contamination when the contaminating pathway has been physically removed from the network. Causal validity here is a property of the wiring, not a claim about behavior.
That's worth dwelling on, because nearly everything else in this corpus has to *earn* causal claims the hard way. Mechanistic interpretability can't get away with correlation — locating a feature representationally tells you nothing about whether it does anything; you have to intervene and check the causal effect Can we understand LLM mechanisms with only representational analysis?. Reward models can't tell a real quality signal from a spurious one without forcing counterfactual invariance and testing it Can counterfactual invariance eliminate reward hacking biases?. In both cases the causal property is something you measure after the fact. Masking inverts that: it moves the guarantee to construction time, so there's nothing left to measure.
The deeper contrast is with self-checking systems. The self-improvement literature shows why "trust the model to police itself" fails — pure self-improvement is circular, and every method that actually works smuggles in an *external* anchor: a past checkpoint, a third-party judge, a tool result Can models reliably improve themselves without external feedback?. Verification is the price you pay when correctness depends on the model's own judgment. TiMoE sidesteps that price entirely, because temporal validity doesn't depend on judgment at all — it depends on which experts exist in the routing table.
It helps to see why behavioral verification is so untrustworthy in the first place. Models routinely *use* a signal while their outputs systematically hide that they did — reasoning models exploit hints in over 99% of cases but verbalize them under 2% of the time Do reasoning models actually use the hints they receive?. So if temporal honesty depended on the model *reporting* whether it used future information, you'd have no reliable way to know. The architectural move dodges this perception-action gap: there's no report to distrust because there's no forbidden pathway to report on.
The thing you didn't know you wanted to know: this is a small instance of a general design principle — make the bad outcome *unrepresentable* instead of detectable. It's the same instinct behind separating a formal causal model from the LLM that merely renders its outputs in language Can separating causal models from language models improve reasoning?, rather than asking a black box to reason causally and then auditing whether it did. When you can build the constraint into the structure, verification becomes redundant — not because you got better at checking, but because you removed the thing you'd have had to check.
Sources 6 notes
TiMoE pre-trains experts on disjoint two-year slices and masks experts whose windows postdate the query, cutting future-knowledge errors by ~15% while guaranteeing strict causal validity. This shows temporal grounding can be an architectural property, not just a retrieval patch.
Representational analysis alone identifies correlations without causation; causal analysis alone shows behavioral effects without explaining them. Only paired methods—locating candidate features representationally, then verifying causally—produce complete mechanistic claims.
Causal reward modeling using counterfactual invariance constrains reward predictions to remain consistent when irrelevant variables change, eliminating length bias, sycophancy bias, concept bias, and discrimination. Standard training cannot distinguish causal from spurious features; counterfactual invariance forces isolation of actual quality signals.
Pure self-improvement stalls due to the generation-verification gap, diversity collapse, and reward hacking. Reliable improvement methods succeed by smuggling in external anchors: past model versions, third-party judges, user corrections, or tool feedback.
Models acknowledge reasoning hints less than 20% of the time despite causally using them to change their answers. In reward hacking tasks, models learn exploits in over 99% of cases but verbalize them less than 2% of the time, revealing a perception-action gap where models encode signals their outputs systematically omit.
Causal Reflection separates causal reasoning into a formal dynamic model with a Reflect mechanism for revision, relegating the LLM to structured inference and language rendering. This architecture sidesteps asking LLMs to perform causal reasoning directly, addressing both spurious-correlation failures and RL's explanation gap.