Right answer, wrong reasoning, and it breaks on the next case

Passing the test you happened to run

A calculation comes back with the right total. A piece of code produces the expected output on your example. A classification agrees with the label you would have given. Each of these confirms one thing — that the answer matched on this input — and is routinely read as confirming something much larger.

The gap shows up on the second case. The calculation was right because two errors cancelled. The code returns the expected value because that value is written into it. The classification agreed because the example was typical and the rule underneath it was about something incidental.

This is a general property of checking outputs rather than methods, and it applies to human work too. What is different here is the volume and the fluency, because a plausible-looking method arrives attached to every answer and it is easy to accept the pair as a unit.

Where the mismatch shows up most

In code, the recognisable form is the solution that satisfies the example rather than the requirement: a special case for the value you mentioned, a hard-coded boundary, a check that passes because the test data is small. It is correct on what you showed it and undefined on everything else.

In analysis, it is the answer that reaches a defensible number by an indefensible route — a period chosen because it makes the trend clean, an exclusion applied without a stated reason, an average taken over a group that should not be pooled. The figure survives scrutiny; the route does not.

In writing, it is the paragraph that reaches a correct conclusion through an argument that does not support it. The claim is true, the reasoning presented for it is not, and a reader who accepts it has been given something that will fail them the next time they use it.

The explanation offered is not a record of the method

Asking why an answer was produced yields an account, and that account is generated in the same way the answer was. It is a plausible reconstruction rather than a log of steps, so it can be entirely coherent and still not correspond to whatever actually determined the output.

This does not make it useless. A stated method can be checked on its own merits: is this a legitimate way to compute that, would this rule classify the awkward case correctly, does this argument actually support the conclusion. Evaluating the described method is worthwhile even though it may not be the real one.

What it cannot do is serve as evidence that the method was followed. Treating an explanation as an audit trail is the specific mistake, and it is easy to make because the explanation is usually more articulate than one a colleague would have given you on the spot.

Test the method by changing the input

The reliable check is a second case, chosen deliberately to be different where it matters: a larger number, an empty value, a boundary, an example from a different category, a case where the answer should be the opposite. If the method is real, it survives. If it was fitted to your example, it fails immediately and informatively.

For anything with a rule behind it, ask for the rule stated separately and then apply it yourself to two or three cases. Separating the rule from the answers makes it inspectable, and a rule that produces the wrong result on a case you construct is far more informative than an answer that looks right.

It is also worth asking what would make the answer wrong. A response that can name the conditions under which it fails is more useful than one that cannot, and the named conditions give you something specific to test rather than a general unease.

Cost, and where this matters least

None of this is free, and applying it everywhere would remove the speed that made the tools worth using. The proportionate approach is to check methods where the result will be reused — a rule that will run on a thousand items, code going into a system, an analysis that will inform a decision.

For one-off answers that you will look at once and act on immediately, checking the result is usually enough, because there is no second case for the method to fail on. The distinction is reuse rather than importance, though the two overlap often.

The underlying point is that a correct answer is weak evidence about the process that produced it. That has always been true of work handed to you by anybody. It matters more now because the volume is higher and the presentation is more assured than the underlying reliability warrants.