What passes between two steps has to carry more than the answer

The interface is the part nobody designs

Breaking a job into steps gets a lot of attention, and rightly, because a compound task that fails as a whole tells you nothing about which part went wrong. What gets far less attention is the payload: the thing step one hands to step two, which is usually whatever step one happened to output.

The default is to pass the answer and nothing else. Step one extracts the relevant passages and passes the passages. Step two judges them and passes a verdict. Step three writes something based on the verdict, and by then everything that would let anybody question the verdict is three steps upstream.

This is a different concern from whether to split at all, which turns on whether the parts need to see each other. Assume the split is right. The question here is what travels along the join.

Carry the source location with the extract

The single most valuable addition to almost any inter-step payload is a pointer back to where the material came from: the document, the page, the section, the row. It costs almost nothing to produce at the point of extraction and it is nearly impossible to reconstruct afterwards.

With it, every later step can be audited. A statement in the final output can be traced to a passage in a source, which is what somebody will ask for the first time a result is challenged, and it is what makes spot-checking a batch possible at all.

Without it, checking means re-running the first step by hand for each item you want to verify, and the practical consequence is that nobody verifies anything.

Pass the ambiguity, not just the resolution

When a step makes a judgement call, the call is what survives and the difficulty that prompted it is what disappears. The extraction step found two candidate dates and chose one; the later steps see a date. The classification step was genuinely unsure; the next step receives a category stated as flatly as any other.

A payload that carries a marker for this — the alternatives, or simply a flag that the item was borderline — changes what the downstream steps and the reviewer can do. Borderline items can be routed to a person, held back, or reported separately, and none of that is possible once the ambiguity has been flattened.

This is also the cheapest form of quality control available in a pipeline, because it requires no additional judgement. Marking difficulty is easier than resolving it correctly, and knowing which fifth of the batch was hard is worth more than a confident value on all of it.

Distinguish absent from not-looked-for

Three states get collapsed into one and cause trouble later: the value was searched for and is not present, the value was not searched for, and the step failed before it got there. Passing all three as an empty field means every later step and every reviewer has to guess which happened.

Keeping them distinct costs a token in the payload and prevents a whole class of misreading, particularly when the output is aggregated. A count of items lacking a field is meaningless if some of those items were never examined.

The same logic applies to anything that was truncated, skipped for length, or processed from a partial source. If a step did less than it was meant to, that fact should travel with its output rather than being discoverable only by inspecting logs nobody reads.

Keep the join simple enough to read

There is an opposite failure, which is a payload that carries so much context that each step is effectively receiving the whole job again. That defeats the purpose of splitting, makes each step harder to reason about, and reintroduces the problem where you cannot tell what a step is responding to.

The useful test is whether a person could look at one payload and understand what the next step is being asked to do with it. If the answer requires reading three earlier stages, the payload is either too thin or too full, and it is worth redesigning before anything is built on top of it.

A short fixed shape, defined in advance and the same for every item, is worth more than a rich one that varies. Steps can then be replaced independently, which is the actual prize, and it is only available when the join was designed rather than inherited.