# A report, not a checkbox

> A verification runs once per state of the tree, and its first output is the answer.

Page: Methodology · Verify
Canonical: https://banes-lab.com/disciplined-methodology/verify#a-report-not-a-checkbox

This section is stop 42 of 102 in the learning route. Previous: [27 - Verify the verifier](https://banes-lab.com/disciplined-methodology/verify/verify-the-verifier.md). Next: [29 - Unknown is not pass](https://banes-lab.com/disciplined-methodology/verify/unknown-is-not-pass.md). It builds on [26 - It looked right](https://banes-lab.com/disciplined-methodology/verify/it-looked-right.md).

A [verification](https://banes-lab.com/records/arch/verification.md) runs once per state of the tree, and its first output is the answer. [Reproducibility](https://banes-lab.com/records/arch/reproducibility.md) is what a second run would test, so a second run that differs is a finding rather than a retry. The output is read whole and never searched for the line you hoped to see. The run writes its report on every exit, so a report being present is never mistaken for a pass. On the grammar page, [validation gates](https://banes-lab.com/pag/validation/validation-gates.md) hold an instruction's claims to the same standard: checkable, and backed by evidence.

### One run, read whole

Re-running a check until it passes turns a verifier into a slot machine. The check fails, is run twice more and passes on the third try, and the flake ships with the change, which is [flaky test normalization](https://banes-lab.com/records/arch/flaky-test-normalization.md) in miniature. A second run of the same state either repeats the answer or reveals non-determinism, and both are already findings from the first run.

For this reason each state of the tree gets one run, its output is read whole, and its report is written on every exit path. The tree is changed between runs, rather than the filter between reads. In practice, the gate runs once and its full output is written to a file. The file is read whole, what it names is fixed, and the tree changes; only then does the gate run again, and never merely to see whether the answer changes. A report's scope is read before its verdict, because a clearance from a check that cannot see the whole surface is worse than a visible gap.

To check this, count the runs per tree state in a session. Anything above one is either a flake that should have been filed or an answer you did not like. A new run is warranted once the tree has changed; the earlier report is then history, and the new run is the measurement. The sign of misuse is a second run whose only difference is the filter.

Reading whole is a rule about structure, not about diligence. A slice of a report answers only the question the reader already thought to ask, while the reason a check writes a finding is to raise something the reader had not thought of. A search is worse than an offset, because an offset shows what was skipped, and a search silently leaves out everything that did not match. The same holds for a file and for a coordination surface. When a read fails because the output is too large, that is the cue to read it in parts until the whole has been read, never the cue to sample it.

A count is evidence of coverage only over the surface the scan reaches. A report therefore carries its derivations as well as its verdict, meaning what it reached, what it excluded and why, so that a green result reads as coverage rather than as silence. A negative result inherits the scope of the query that produced it and carries no evidence of its own, so before reporting that something does not exist, the measurement is run again one scope wider, over every surface where the thing could be declared.

## Links to

- [Verification](https://banes-lab.com/records/arch/verification.md)
- [Reproducibility](https://banes-lab.com/records/arch/reproducibility.md)
- [Validation gates](https://banes-lab.com/pag/validation/validation-gates.md)
- [Flaky Test Normalization](https://banes-lab.com/records/arch/flaky-test-normalization.md)

## Linked from

- [PAG and the method](https://banes-lab.com/pag/introduction/pag-and-the-method.md)
- [Document structure](https://banes-lab.com/pag/guide/document-structure.md)
- [Validation gates](https://banes-lab.com/pag/validation/validation-gates.md)
- [Planning templates](https://banes-lab.com/pag/templates/templates-planning.md)
- [Detect, log, fix](https://banes-lab.com/disciplined-methodology/build/detect-log-fix.md)
- [The honest gaps](https://banes-lab.com/disciplined-methodology/ship/the-honest-gaps.md)