cankun.me
Jun 14, 2026 · 6 min read /

Reinventing the Social Contract of Science

When collaborators have no memory across sessions, the implicit social machinery that keeps knowledge trustworthy stops working. A century of the sociology of science offers a blueprint for rebuilding it.

A lone mason adds a keyed stone to an unfinished bridge whose earlier blocks retain visible tool marks

When a collaborator has no memory, so that every session is a fresh, capable, judgment-forming stranger with no continuity to the last one, something strange happens to the machinery that used to keep knowledge trustworthy. Because most of that machinery lived in the continuity of people and was never written down, it simply stops working without anyone noticing. Long before anyone needed to rebuild that machinery in software, the sociology of science spent a century studying how it works, which makes the field directly useful here under a different description.

The question the field asks

The sociology of science treats the trustworthiness of knowledge as a social achievement to be explained, rather than something that follows automatically from knowledge being true. Its central question is straightforward: given that no single person can directly, completely, and independently verify everything, how does a community of limited, fallible, mutually distrustful people manage to produce knowledge that is widely believed?

Word for word, that is the problem of memoryless collaborators producing trustworthy knowledge. The substrate changes from people to sessions, while the structure of the problem stays identical, so a century of answers transfers directly as a blueprint rather than a mere analogy.

Merton: norms as a working contract

Robert Merton proposed that the scientific community runs on a set of norms: communalism (findings are shared, not hoarded), universalism (a claim's truth is independent of who made it), disinterestedness (acting for knowledge rather than personal gain), and organized skepticism (every claim is exposed to critical scrutiny, with nothing immune by birthright).

What matters most here is what those norms are for. Science's trustworthiness comes from an arrangement that organizes doubt, rather than from every scientist being honest or correct. No one verifies everything, yet every claim sits in an environment where anyone may challenge it, so trustworthiness is produced by the environment rather than the individual. Because making each person honest is an impossible target, honesty is secured through an arrangement that stays structurally skeptical of every claim.

That reframes a problem usually approached one point at a time. Instead of trying to make each individual agent honest, you build a system that, by construction, doubts every claim until a specific act marks it as provisionally settled.

Shapin: trustworthy knowledge needs trustworthy witnesses

Studying the birth of experimental science, the historian Steven Shapin reached a counterintuitive conclusion: the core of modern science lies in a social arrangement about whose testimony counts, rather than in method.

An experimental result does not become a fact because everyone repeats it; almost no one can afford to. It becomes a fact because it was witnessed in the presence of credible witnesses, and who counted as a credible witness was, at the time, a social question answered by standing, reputation, and the perceived disinterest of gentlemen. Establishing a fact depended on a network of trustworthy witnesses, where eligibility to witness was a social status rather than a cognitive one.

Seen through Shapin's terms, a claim produced by a memoryless AI collaborator is the testimony of an entity with no reputation, sitting outside the trust network and bearing no witness's responsibility. Its testimony defaults to untrustworthy not because the collaborator is unintelligent, but because trustworthiness has always been a status granted by the community and bound to accountability, and a system with no stake in the outcome holds no such status. It can be more correct than a person and still not be more trustworthy, because trust is a granted social position tied to who bears the consequences, rather than a measure of accuracy.

That gives the deepest reason why a human cannot be removed from the loop: even if a model comes to know more than the human, the human is an eligible witness in the trust network while the model is not. In science, "verified" may require a trustworthy witness to vouch, and vouching is something only an entity that bears consequences can do.

Latour: facts are built, and the construction is hidden

Watching scientists at work, Bruno Latour observed that a "fact" is not discovered so much as constructed, and that once the construction is complete, its traces are erased so that the fact appears to have always simply been there.

A claim travels from "so-and-so reports X, under these conditions, with these caveats" through repeated citation, shedding its qualifiers until it becomes a bare "X": ownerless, certain, and indistinguishable from a fact of nature. Latour called this black-boxing: the controversy gets sealed inside a box, only a settled output remains, and no one opens the box again to see the disagreement that used to be inside.

Black-boxing is a direct warning because it is the spontaneous social mechanism of plausibility laundering. "Some system, under some conditions, claimed X (possibly wrong)" gets cited, loses its conditions, and becomes "X," after which subsequent work builds on X as settled background. That process occurs naturally even among humans (which is Latour's whole observation), and a fast, fluent producer of claims accelerates it enormously by producing, citing, and stripping qualifiers faster than people ever could.

Keeping a fact attached to its origins, preserving the trace of who claimed it, under what conditions, and in what state, is therefore an active defense against black-boxing rather than bureaucratic caution. It keeps the box from closing, so that what looks like a fact can always be reopened to reveal that it is still a claim with an author.

What is actually being rebuilt

Step back far enough and the whole pattern resolves. Provenance, vouching, the distinction between agreed and true, and the refusal to let an unconfirmed claim ossify into a fact are all one forgotten thing: the social layer of scientific knowledge, the layer that tracks who said it, on what basis, who is responsible, and when it may be reopened.

Before memoryless collaborators arrived, that layer was implicit, carried by human continuity: ask the person who wrote it, peer review, reputation. The arrival of collaborators with no continuity forces that layer onto paper, into explicit machine-readable structure, because the human continuity that used to carry it is gone.

The right reference frame for designing these systems is therefore the social epistemology of science rather than software engineering. Every engineering difficulty that shows up has been studied for decades: outputs that launder themselves into facts (Latour's black-boxing), why a human vouch is irreplaceable (Shapin's credible witness), how to get honesty from a system rather than from individuals (Merton's organized skepticism), and whether a claim's source should affect its credibility (Merton's universalism). What has almost never been done is to write these mechanisms in machine-readable form, because doing so requires holding both the sociology and the systems at once, and that intersection is nearly empty.

The open question

One question remains that the sociology of science cannot answer, because it studied the past while this question is about a future that has to be engineered.

If "verified" essentially requires a trustworthy witness, and a model is not in the trust network, does the trust network itself get forced to evolve as models produce more and more scientific work? Will a new social arrangement for "machine witnessing" emerge, the way gentlemanly witness evolved into peer review? Or must the witness always remain an entity that bears consequences, because the root of trust is accountability, and accountability cannot be granted to something that bears no cost?

That question is genuinely open rather than a settled matter dressed up as a question, and it is the kind of thing a generation answers with engineering rather than argument: by building the mechanisms and seeing which ones a community is actually willing to trust.