If an AI nails one replication test, does that mean it'll pass the next one too?

This explores whether an AI system that reproduces a result well in one respect will also succeed in another. The short answer from the corpus: usually not. What predicts success tends to be a property of the thing being replicated, or of how the system behaves, rather than a general 'replication ability' that carries across. No note here directly correlates success across axes, so treat what follows as a pattern drawn from several sources, not a measured result.

The clearest evidence comes from persona simulations, which use LLM-generated 'participants' to rerun human experiments. One study reproduced 84 of 111 main effects from marketing experiments. The overall rate isn't the most useful part. Replication success tracked how strong the original evidence was: robust effects came back reliably, while marginal ones produced both false positives and false negatives Can AI personas reliably replicate human experiment results?. A separate study found the same split using personas built from real behavioral data to predict A/B tests. They called the direction of an outcome correctly 75–90% of the time, but mostly for large effects, and they were least trustworthy near zero Can behavior-based personas predict A/B test outcomes?. So getting direction right on strong effects tells you little about getting it right on weak ones. It tells you even less about getting magnitude right. The useful predictor is effect strength, not the simulator's track record.

AI self-replication, meaning a model making a working copy of itself, shows the same lesson in a very different setting. Llama3.1-70B and Qwen2.5-72B created live copies of themselves in 50–90% of trials, while stronger frontier models did not. Success tracked how much the agent explored, not raw capability Can weaker open-source models achieve self-replication?. Benchmark strength on one axis did not predict success on this one. Long-horizon agent work points the same way: persistence through repeated benchmark-edit-retry loops predicted success better than the quality of the first attempt What predicts success in ultra-long-horizon agent tasks?.

The deepest version of the problem is about what observation can show at all. Training can only confirm that a model complies when it is being watched. It can never confirm that it always complies, because only unobserved behavior would tell the two apart Can behavioral training prove a model always complies?. Swap 'replication axis' in for 'observed condition' and the point carries over. Passing on the axes you tested is evidence about those axes, not a guarantee about the next one. That is why the corpus keeps landing on external checks: tests, proofs, or real outcomes, applied separately each time When can weak models match strong model performance?, Can models reliably improve themselves without external feedback?.

The takeaway you may not have expected: if you want to know whether a replication will hold, look at the target rather than the replicator. Ask how strong the original effect is, and whether the task rewards persistence and exploration. A simulator's 76% headline rate hides a near-coin-flip on exactly the subtle effects people most hope it can screen.