cankun.me
Jun 14, 2026 · 6 min read /

Querying a Black Box

Debugging a system whose source code exists versus interrogating one with no source code at all: Hume, Popper, and why automation flies on one and sinks into quicksand on the other.

An observer probes a sealed black chamber while a faint distorted trace emerges from the far side

Some kinds of work come down to debugging a system whose source code exists. Other kinds require interrogating a system that has no source code at all, one you can only poke from the outside and that answers you through an instrument that may itself be lying. From the inside, the two situations feel much alike, since both involve uncertainty and careful reasoning toward a conclusion. Epistemically, though, they are very different, and that difference explains almost everything about why automation works beautifully on one and finds the other to be quicksand.

It is worth building the distinction slowly, because the philosophy that illuminates it is old, and the original version is sharper than the casual summary that usually gets passed around.

Hume's problem, in its strong form

The problem of induction is usually told softly: you have seen a thousand white swans, yet you cannot conclude that the next one will be white. Told that way, it sounds like a gentle warning against overgeneralizing. Hume's actual version is much harder.

Why do you believe the future will resemble the past at all? If you answer "because the past has always resembled the past," that is itself an inductive inference, using induction to justify induction, which is circular. If you answer "because nature is uniform," why believe nature is uniform? Only because it has been so far, which is induction again, and circular again. Hume's conclusion goes beyond saying that induction is somewhat unreliable: he argues that induction has no rational justification at all. As a matter of logic, every "the sun will rise tomorrow" and every "this drug will still work next week" is just as groundless as its denial. We make these inferences anyway, Hume says, out of habit rather than reason, as a psychological fact rather than a logical entitlement.

This matters because the entire enterprise of reasoning from data to conclusion rests on the step Hume dissolved. When you move from "in my sample, X and Y are associated" to "X and Y are really associated," there is no logical bridge between the two. The work is not useless, but logical validity was never what held it up, and the next two centuries of philosophy were spent arguing about what does hold it up instead.

Popper's escape, and what it costs

Popper accepts that Hume won and that induction has no logical justification, then argues that the mistake was thinking science runs on induction in the first place.

You can never verify a universal claim by observation, because the next observation might break it, yet you can falsify it with a single counterexample. Verification and falsification are logically asymmetric: no number of white swans confirms "all swans are white," whereas one black swan refutes it. Science, for Popper, is therefore the proposal of bold, falsifiable conjectures and the determined attempt to kill them, rather than the steady accumulation of supporting evidence until you feel sure; whatever survives the attempt is kept only provisionally. A theory's scientific status depends not on how much evidence supports it, but on whether it sticks its neck out by stating in advance what observation would prove it wrong.

An important consequence follows from this. For a conjecture to have any falsifying power, the prediction must come before the evidence: you have to state what you bet you will see, and only then look. If you look first, see the result, and then construct an explanation, you have not tested anything, because any result can be fit with some story after the fact, and a story that explains everything has been refuted by nothing. That temporal order is the whole difference between a test and a rationalization, rather than mere bookkeeping.

Why falsification is never clean

Popper's account is beautiful, and it is also incomplete in a way that has a name: the Duhem-Quine problem. When an experiment contradicts a theory, what exactly has been falsified?

A prediction never comes from a theory alone. It comes from the theory plus a large cloud of auxiliary assumptions: that the instrument works, the reagents are pure, the sample is clean, the statistical model is appropriate, and that one confounder you never thought of is absent. When the result conflicts with the prediction, logic tells you only that something in the whole bundle is wrong, without telling you which part. You can therefore always rescue the theory you love by blaming an auxiliary assumption ("must be the equipment, run it again"). Falsification is never the clean single stroke Popper imagined, because a black swan can always be explained away as "not really a swan" or "I misread."

In domains where the instruments are transparent, this rarely bites. When you read a variable in a debugger, the debugger does not lie, so a failed test usually means the code is wrong and blame is easy to locate. Where the instrument is itself opaque, though, and the measuring apparatus is another black box you do not fully understand, blame for a contradiction cannot be cleanly assigned. You set out to interrogate one black box only to find that your interrogation tool is another.

Map and territory

All of this fits into a single picture. Every model, claim, or hypothesis is a map, while the territory is the real world that exists independently of any map. Hume's problem, Popper's asymmetry, and the Duhem-Quine undecidability all flow from one fact: we only ever have maps to compare against other maps, and we never get to step outside all maps to grab the territory directly and check.

Whenever you try to verify a claim, you do it with another observation, which is itself a map mediated by your instrument, your method, and your conceptual frame. You remain on the map side, and the territory intervenes only in the rare places where it is forced to answer directly: a genuine experiment, and above all an experiment that puts a question to the irreducible complexity of the real world over real time. Outside those rare moments when the territory talks back, what we take for a fact is one map agreeing with another.

The unifying frame: querying a black box

Put these pieces together and the distinction from the beginning becomes precise. Some work is debugging a white box: a system with source code, whose state you can read directly and whose measurements are transparent. Other work is querying a black box: a system with no source code, one you cannot read directly and can only perturb, whose every answer comes back through a noisy, possibly deceptive instrument, and which never promised that its underlying rules are simple or stable.

This single frame explains the whole asymmetry between the two kinds of work. One has a cheap verifier and the other does not because a white box's state can be read directly and compared, whereas a black box's state can only be perturbed and inferred through noise. Automation soars on one and stalls on the other because querying a white box is instant, certain, and infinitely repeatable, while querying a black box is slow, noisy, and never exactly reproducible. Plausibility is far more dangerous in the black box because a plausible error in a white box is quickly exposed by transparent measurement, whereas in a black box it can hide indefinitely behind "maybe it was the instrument." And the deepest verifier remains irreducible because no cheaper query can substitute for querying the ultimate black box, the real world, in its full diversity over real time; every cheaper query is just a different black box, separated from the real one by another map-territory gap that cannot be cleanly crossed.

This frame does not make the problem go away; nothing does, because that is simply what it means to live on the map side. What the frame offers is clarity about which kind of problem you are actually in, which is the prerequisite for not lying to yourself about what your conclusions are worth. The discipline that follows is easy to state and hard to keep: never let "I have a map of it" pass for "I have touched the territory," because even when the map is excellent, the territory still has not spoken.