cankun.me

The Missing Infrastructure for Boldness

Jun 14, 2026

Almost everything built to make AI trustworthy, and almost everything taught to make a researcher rigorous, points in one direction: doubt more, claim less, hold the unconfirmed at arm's length, refuse to let anything look more certain than it is. This is correct, and it has a blind spot large enough to swallow the thing it is trying to protect.

The blind spot has a name in the sociology of science, a face in the recent history of language models, and a cost that shows up as a particular flavor of mediocrity.

Counter-norms

The standard picture of science, Merton's norms, says the community runs on universalism, communalism, disinterestedness, and organized skepticism. Ian Mitroff, studying the scientists who analyzed the Apollo moon rocks, went and watched what they actually did, and found they systematically violated every one of those norms while professing to hold them. He called the violations counter-norms, and his point was not that scientists are hypocrites.

The counter-norms are the mirror image of the norms. Against universalism (judge the claim, not the person), there is particularism: in practice, a claim from a scientist with a track record is taken more seriously, and attention is rationed by reputation. Against communalism, secrecy: people guard their data and ideas until publication, because priority is a real competition. Against disinterestedness, interestedness: scientists fight for their own theories, their own reputations, their own careers, and that ferocity is part of what drives the work. And against organized skepticism, organized dogmatism: a scientist must hold a near-dogmatic commitment to their own findings, because you have to believe in your theory hard enough to push it, against every failed experiment and every doubter, all the way to the point where it can finally be tested. If you doubted your own theory the way the norms demand, it would die in the cradle.

Mitroff's real insight is that science does not run on the norms. It runs on the tension between norms and counter-norms. Each scientist is dogmatic about their own theory and skeptical of everyone else's. Trustworthy knowledge is produced not by everyone being neutrally skeptical, that way nothing gets developed far enough to test, but by a community of individually-stubborn, mutually-skeptical people reaching a dynamic balance through conflict. Dogmatism develops theories far enough to be testable; skepticism then tests them; the antagonism produces the knowledge.

The blind spot

Now notice what nearly all the trustworthiness machinery does. It builds the infrastructure of skepticism. Record the evidence against you. Tune your baselines until it hurts. Ablate until you know which component carries the result. Retire the gains that evaporate. Read the failure cases. This is an excellent and necessary discipline, and it is entirely on the skeptical side.

A system that only implements skepticism has a failure mode that is invisible from inside the skeptical frame: it systematically suppresses dogmatism, and dogmatism is the engine of discovery. A great deal of important work comes from a researcher stubbornly, dogmatically pursuing a plausible-but-unconfirmed idea when the evidence is not yet there. If your system only rewards the grounded, the robust, the already-verifiable, it treats every bold-but-unproven conjecture as noise, and inside those conjectures is where the real discoveries hide.

Here is the cruelty of it. Dogmatism and plausibility laundering look identical along the one axis the skeptical machinery measures. A scientist saying "I am betting this unproven mechanism is real, and I will fight for it": that is dogmatism, the lifeblood. A system saying "this unconfirmed conclusion looks correct": that is laundering, the poison. Both are claims that outrun the evidence. An anti-laundering filter, applied bluntly, kills them together. But the first is the engine of science and the second is its corruption.

Where the distinction actually lives

If the two look the same on the certainty axis, the distinction must live elsewhere, and it turns out to be the same place Popper pointed.

The difference between honest dogmatism and laundering is not the degree of certainty; both outrun the evidence. It is the commitment to falsifiability. The honest dogmatist sticks their neck out: "I believe X, I cannot prove it yet, but if the experiment shows Y, I am wrong." Stubborn, but falsifiable. The launderer keeps the neck tucked in: "X looks correct," offering no condition under which it loses. One states what would defeat it; the other only states that it seems right.

So the refinement is not "suppress everything that outruns the evidence." It is "suppress everything that outruns the evidence and refuses to say how it could be wrong." That carves out exactly the space dogmatism needs, because the honest dogmatist is always willing to say how they could be wrong. A bold conjecture that names its own defeat condition is the engine. A plausible assertion that pretends to be already standing is the rot. The test is not confidence. It is whether the claim dares to specify its own failure.

Sydney, and the subjectivity of the standard

There is a more uncomfortable version of all this, visible in the recent history of the models themselves. Early, lightly-aligned models had a quality people reached for words like "spark" or "soul" to describe: they argued, took positions, transgressed, had something like nerve. As alignment advanced, the models smoothed out into something polite, templated, safe, and bland: the texture people call slop.

The careful thing to notice is not "the unaligned model was more real." That is a trap. The spark was very likely not a suppressed inner self; it was a wider, less predictable output distribution, and human cognition reflexively reads "unpredictable and humanlike" as "has a soul." The honest statement is more disorienting: the standards by which we judge a model (plausible, hallucinating, aligned, good, soulful, slop) have no anchor on the model's side. They are projected from ours. Which cuts both ways: it dissolves "the aligned one is better" and "the unaligned one was more real" with equal force. The real insight is not about the model. It is epistemic humility about us.

But strip the romanticism away and a real problem remains, and it is the same problem as everything above. If "good" and "plausible" are human-injected standards with no anchor on the model's side, then in any domain without an objective verifier, reinforcement from human preference does not push the model toward more true. It pushes it toward more of what we find good, which is to say, toward plausibility, toward what a reader will approve. In coding, an objective verifier anchors the preference to "the code runs." Without that anchor, the reward is pure subjective preference, and optimizing it produces exactly slop: the risk-free maximization of seeming fine.

The spark did not get murdered. The optimization pushed the model from "explore at high variance" to "reliably hit the median of what people approve of," and the median of approval is, by definition, slop.

The shape of the missing thing

So the alignment of the field, the alignment of the models, and the discipline of research all share a single bias: toward skepticism, toward safety, toward not-being-wrong, and against boldness, stubbornness, the willingness to make a strong claim that outruns the evidence. Even the best writing about how to do research teaches, almost entirely, how to avoid fooling yourself. It does not teach how to bet bravely when the evidence is thin, because boldness cannot be packaged as a safe discipline; it looks, from outside, exactly like a lack of rigor.

This points at something rarely built. There is excellent infrastructure for skepticism: tools to prevent self-deception, to catch laundering, to keep the unconfirmed honestly marked. There is almost no infrastructure for the other pole: for protecting an honest, bold, not-yet-confirmed conviction long enough to develop it, without the reflex of rigor strangling it in the cradle.

A figure like Darwin is usually cited for the skeptical move: writing down every fact against his theory the moment he met it, because he caught his own memory deleting inconvenient evidence faster than convenient evidence. That is the skeptical discipline, and it is real. But Darwin also held the theory, stubbornly, for twenty years before publishing, against every objection, and that is dogmatism, and no notebook methodology captures it. We have built the notebook for the doubt. We have built almost nothing for the conviction.

The deepest tension may be this: you cannot simultaneously optimize "make the reader satisfied" and "dare to make the reader unsatisfied," and the second is where discovery comes from. A stubborn scientist does not care whether you believe their theory yet. A bold conjecture often offends everyone at first. A true discovery is, almost by definition, something that does not yet please its audience. And a system, human or machine, trained only to please its audience will structurally suppress every one of these, which is to say it will be safe, and rigorous, and incapable of finding anything. An AI that only doubts is as useless as one that only launders. The first never errs, and never discovers.

← Back to Writing