cankun.me
Jun 14, 2026 · 7 min read /

The Garden, the Multiverse, and What History Cannot Teach

Turning the garden of forking paths into a resource gives us robustness without a verifier, right up to the wall it eventually runs into.

A formal garden of branching paths ends at a blank wall, beyond which the crown of a wild tree rises

At the heart of data analysis sits a curse that, once you see it clearly, contains its own partial cure; and when you push that cure far enough, it runs into a wall that is a fact about what discovery is rather than an engineering limit. Walking the whole arc is worthwhile because it connects a famous statistical worry to the question of what a machine could ever be trained to do.

The garden of forking paths

The worry is Andrew Gelman's, and its power lies in how it survives every defense you would normally raise. The naive concern about untrustworthy results is that someone went fishing, trying analysis after analysis until something crossed the significance threshold. The forking-paths argument is that even a completely honest analyst, who fixed their hypothesis in advance and ran exactly one analysis, can produce an untrustworthy result.

This happens because the analysis is full of small decisions made after seeing the data: how to filter outliers, which normalization to use, how to group, whether to drop a sample, which model to choose, and where to set a threshold. Each decision is reasonable on its own, yet each is data-dependent: had the data looked different, the analyst would have decided differently. Because of that, the paths not taken still bear on the credibility of the result. Without any conscious cheating required, the analyst walked one route through a garden of thousands of branching paths, believing it was the only sensible route, when in fact a slightly different dataset would have sent them down another branch to a different conclusion. Every untraveled branch is an invisible deduction from how much the single result should be believed.

Turning the curse into a resource

Traditionally, this has been a curse precisely because you cannot walk all the paths when there is only one analyst, one lifetime, and one route. That constraint is the exact one that cheap, capable agents dissolve. When running an analysis is nearly free, you can actually walk most of the paths. Instead of choosing one pipeline and praying it is robust, you can run the field's whole space of reasonable pipelines and observe how a conclusion behaves across them.

This is the industrialized form of the cure: turning the garden of forking paths from an enemy into a resource. Rather than treating a conclusion as a binary that holds or does not, you can give it a robustness profile, a map of how stable it is across the space of tool choices, parameter settings, and data subsets: whether it survives switching the differential-expression method, moving the threshold, or bootstrapping the data. That profile is rich, and, crucially, it requires no verifier. Because you only need to observe the distribution of the conclusion across all reasonable pipelines without knowing which pipeline is correct, the approach works in domains with no cheap ground truth, where compute is abundant and truth is scarce.

Three things robustness can reward, and which two matter

Once you can run the whole space, you can extract a training signal from it, though there are three different things you could reward and they are not the same.

First, you could reward robust conclusions directly, giving a high score to a result that holds across ninety percent of reasonable pipelines. While this trains the model to produce robust conclusions, it also punishes true-but-fragile findings. Some real discoveries are visible only under one specific method, because only that method had the power to see them. Rewarding convergence breeds a conservative parrot that only reports what holds no matter how you look, which is the same as only reporting the bland.

Second, you could reward honest calibration: not "the conclusion is robust," but "the model's stated confidence in its conclusion's robustness was accurate." When the model claims something is robust, you run the multiverse to check, rewarding it if it was right and penalizing it if it overclaimed. This trains the model to know honestly how fragile its own conclusions are. Instead of punishing fragile discoveries, it punishes calling a fragile thing robust, turning the anti-laundering principle into a reward.

Third, you could reward noticing disagreement. When two pipelines diverge, that divergence is information rather than noise, pointing at a methodologically sensitive spot that might be real biological heterogeneity or might be one tool's artifact. A good system notices and explains the divergence rather than silently picking one. That turns disagreement into a source of value, specifically into a guide to which next experiment would resolve it.

The calibration reward and the disagreement reward are the ones worth having because they train honesty and judgment, whereas the convergence reward is a trap that trains a coward. All three are free of any verifier; they need only the multiverse and the compute to run it.

The trap inside the cure

Still, there is a flaw at the heart of all this, the same one the map-territory picture predicts: "All the mainstream tools agree" does not mean "close to the truth."

The field's mainstream tools share enormous amounts of structure, including the same assumptions, the same statistical frame, and sometimes the same underlying bugs. When they agree, it may be because they are wrong in the same way rather than because the conclusion is right. If the entire field's methodology assumes some false premise, then "runs through all the mainstream tools and stays stable" yields a high robustness score for a conclusion that is wholly wrong. This is the map-territory gap in its local form: walk all the maps, and you get the maps' consensus rather than the territory. Multiverse agreement verifies stability within the field's methodological consensus, not truth, and the gap between those two is the exact space in which an entire field can be collectively mistaken.

Because of that, the robustness signal must honestly mark itself as "stable within methodological consensus": agreed, not verified. The only thing that closes the gap is the territory talking back through rare ground-truth anchors, used not as a dense training signal but to calibrate how well consensus-robustness actually tracks truth. Discovering that gap, measuring how often a high-robustness conclusion later proves false, is itself among the most valuable things one could produce, because it quantifies the distance between the consensus of maps and the territory.

What history can and cannot teach

There is a seductive way to manufacture a verifier for free: use past scientific discoveries whose answers history has already revealed. You give a model the situation a scientist faced before the discovery, let it infer, and reward it against the answer that was later confirmed, replacing a slow, expensive verifier with one history has already run. Clever as this is, it has two failure modes of very different severity.

The shallow one is answer leakage. Because the model read every textbook in pretraining, it does not infer the answer, it retrieves it, and the reward then trains "remembering" rather than "inferring." This is partly fixable: you can use very new or unpublished findings the model could not have seen, reward the quality of the reasoning path rather than the answer, or, most interestingly, use historical cases that were later overturned and reward identifying their fragility, the experiment that would break them, rather than reproducing the old conclusion. That last move turns leakage from a bug into a feature: the model knows the later truth, yet it never memorized how to recognize, at the time, that the old consensus would fall, and that recognition cannot be retrieved.

The deep failure mode cannot be fixed, and it is the more important one. Hindsight reshapes the question as well as revealing the answer. Before the discovery, the scientist faced an un-conceptualized mess, not knowing which variables to measure, what to ask, or what was even relevant; the discovery revealed, simultaneously, how to see the problem. When you construct the training environment today, you stand on the far side of the answer, so the data you hand the model, the variables, and the very framing of the task already leak the discovery. The hardest part, realizing what to ask and inventing the new concept the old frame could not hold, has been pre-solved by your environment, leaving the model to fill in only the last step on a stage you already set.

This points to the unfixable wall that everything else in this terrain runs into. You can train inference within an already-correct framing, but you cannot train the creation of a new framing, because training requires ground truth, and the ground truth for a new frame does not exist until the frame has been created. Once the answer is known, the new concept already exists, and the act of discovering it has vanished. Rather than an engineering limit, this is the logical structure of discovery: the irreducible part of a real discovery, realizing the old vocabulary cannot hold what you are seeing, is the exact part no environment with a known answer can ever contain.

The corner that remains

What survives all of this is narrow and, for that reason, valuable. The overturned consensus of the past is a badly underpriced resource, because its reward, was later refuted, is a definite historical fact, while the capability it would train, recognizing, at the time, that it would be refuted, cannot be gotten by retrieving an answer. You cannot train a machine to reproduce science's successes without it cheating, yet you might be able to train it to reproduce science's self-correction; and self-correction, the organized skepticism that is the real source of trustworthy knowledge, is the one thing history offers in enormous, well-labeled supply. The goal worth aiming at may therefore be a machine that doubts well rather than a machine that discovers, and of doubt, done honestly and then vindicated by history, there is no shortage of training data at all.