cankun.me

Reinventing the Social Contract of Science

Jun 14, 2026

When a collaborator has no memory, when every session is a fresh, capable, judgment-forming stranger with no continuity to the last one, a strange thing happens to all the machinery that used to keep knowledge trustworthy. It stops working silently, because most of that machinery was never written down. It lived in the continuity of people. And the discipline that studied this machinery, before anyone needed to rebuild it in software, was the sociology of science.

It is worth taking seriously, because it turns out to have been working on exactly this problem for a century, under a different description.

The question the field asks

The sociology of science treats the trustworthiness of knowledge as a social achievement to be explained, rather than as something that follows automatically from knowledge being true. Its central question is: given that no single person can directly, completely, independently verify everything, how does a community of limited, fallible, mutually-distrustful people manage to produce knowledge that is widely believed?

This is, word for word, the problem of memoryless collaborators producing trustworthy knowledge. The substrate is different, people in one case, sessions in the other, but the structure is identical. Which means a century of answers transfers directly. Not as analogy. As blueprint.

Merton: norms as a working contract

Robert Merton proposed that the scientific community runs on a set of norms: communalism (findings are shared, not hoarded), universalism (a claim's truth is independent of who made it), disinterestedness (acting for knowledge rather than personal gain), and organized skepticism (every claim is exposed to critical scrutiny; nothing is immune by birthright).

The crucial insight is what these norms are for. Science's trustworthiness does not come from every scientist being honest or correct. It comes from an arrangement that organizes doubt. No one verifies everything, but every claim sits in an environment where anyone may challenge it, and trustworthiness is produced by the environment, not the individual. Honesty is not secured by making each person honest, an impossible target. It is secured by a structure that is structurally skeptical of every claim.

This reframes a problem that is usually approached one point at a time. You do not get honesty by making each agent honest. You get it by building a system that, by construction, doubts every claim until a specific act marks it as provisionally settled.

Shapin: trustworthy knowledge needs trustworthy witnesses

The historian Steven Shapin, studying the birth of experimental science, reached a counterintuitive conclusion: the core of modern science is not method but a social arrangement about whose testimony counts.

How does an experimental result become a fact? Not because everyone repeats it; almost no one can afford to. It becomes a fact because it was witnessed, in the presence of credible witnesses. And who counts as a credible witness was, at the time, a social question, answered by standing, reputation, the perceived disinterest of gentlemen. The establishment of a fact depended on a network of trustworthy witnesses, and eligibility to witness was a social status, not a cognitive one.

This is the sharpest possible light on the role of a memoryless AI collaborator. In Shapin's terms, a claim produced by such a collaborator is the testimony of an entity with no reputation, outside the trust network, bearing no witness's responsibility. Its testimony defaults to untrustworthy, not because it is unintelligent, but because trustworthiness has always been a status the community grants, bound to accountability, and a system with no stake in the outcome holds no such status. It can be more correct than a person and still not be more trustworthy, because trust is not a measure of accuracy. It is a granted social position, tied to who bears the consequences.

This gives the deepest version of why a human cannot be removed from the loop: not because the human knows more, a model may come to know more, but because the human is an eligible witness in the trust network and the model is not. "Verified," in science, may require a trustworthy witness to vouch, and vouching is something only an entity that bears consequences can do.

Latour: facts are built, and the construction is hidden

Bruno Latour, watching scientists at work, observed that a "fact" is not discovered so much as constructed, and that once the construction is complete, its traces are erased, so the fact appears to have always simply been there.

A claim travels from "so-and-so reports X, under these conditions, with these caveats" through repeated citation, shedding its qualifiers, until it becomes a bare "X", ownerless, certain, indistinguishable from a fact of nature. He called this black-boxing: the controversy gets sealed inside a box, only a settled output remains, and no one opens it again to see the disagreement that used to be inside.

This is a direct warning, because black-boxing is the spontaneous social mechanism of plausibility laundering. "Some system, under some conditions, claimed X (possibly wrong)" gets cited, loses its conditions, becomes "X," and subsequent work builds on X as settled background. The process occurs naturally even among humans, that is Latour's whole observation, and a fast, fluent producer of claims accelerates it enormously, because it produces, cites, and strips qualifiers faster than people ever could.

So the work of keeping a fact attached to its origins, keeping the trace of who claimed it, under what conditions, in what state, is not bureaucratic caution. It is the active prevention of black-boxing. It keeps the box from closing, so that what looks like a fact can always be reopened to reveal that it is still a claim with an author.

What is actually being rebuilt

Step back far enough and the whole pattern resolves. Provenance, vouching, the distinction between agreed and true, the refusal to let an unconfirmed claim ossify into a fact, all of it is one forgotten thing: the social layer of scientific knowledge, the layer that tracks who said it, on what basis, who is responsible, and when it may be reopened.

Before memoryless collaborators, this layer was implicit, carried by human continuity: ask the person who wrote it, peer review, reputation. The arrival of collaborators with no continuity forces the layer onto paper, into explicit machine-readable structure, because the human continuity that used to carry it is gone.

Which means the correct reference frame for designing these systems is not software engineering. It is the social epistemology of science. Every engineering difficulty that shows up: outputs that launder themselves into facts (Latour's black-boxing), why a human vouch is irreplaceable (Shapin's credible witness), how to get honesty from a system rather than from individuals (Merton's organized skepticism), whether a claim's source should affect its credibility (Merton's universalism), each has been studied for decades. What has almost never been done is to write these mechanisms in machine-readable form, because doing it requires holding both the sociology and the systems at once, and that intersection is nearly empty.

The open question

There is a question here that the sociology of science cannot answer, because it studied the past, and this question is about a future that has to be engineered.

If "verified" essentially requires a trustworthy witness, and a model is not in the trust network, then as models produce more and more scientific work, does the trust network itself get forced to evolve? Will a new social arrangement for "machine witnessing" emerge, the way gentlemanly witness evolved into peer review? Or must the witness always remain an entity that bears consequences, because the root of trust is accountability, and accountability cannot be granted to something that bears no cost?

This is not a settled matter dressed up as a question. It is genuinely open, and it is the kind of thing a generation answers with engineering rather than argument, by building the mechanisms and seeing which ones a community is actually willing to trust.

← Back to Writing