What Makes a Quantitative Claim Credible?

Direct answer: A quantitative claim becomes credible when a reader can inspect how it was produced and what would make it fail. The minimum chain is a declared question, traceable data, pre-specified failure criteria, tested implementation, quantified uncertainty, reproducible artifacts, and a conclusion bounded to the decision and sample. Those elements support justified confidence—not permanent truth. A polished chart, a large backtest, or a precise number cannot substitute for that chain.

Credibility is an inspectable chain#

Quantitative work often earns trust from its surface. More decimal places look precise. A complex model looks sophisticated. A long sample looks comprehensive. None of those features answers the more important question: can another reader trace the claim from the decision back through the data, transformations, implementation, and test?

Credibility is therefore not one statistic. It is a connected record.

Break one important link and the conclusion may still be interesting, but it becomes harder to rely on.

The mechanism: error enters before the model runs#

Model risk is often treated as a problem inside an equation. In practice, uncertainty enters earlier and persists later.

Selection determines which observations exist. Definitions determine what each field means. Transformations determine which information reaches the estimator. Parameter choices determine what the model can express. Validation determines which failures become visible. Interpretation determines whether a narrow result stays narrow.

That is why governance belongs inside quantitative research rather than after it. The U.S. banking agencies' 2026 Revised Guidance on Model Risk Management connects a model's purpose and intended use to development, testing, validation, monitoring, governance, documentation, and effective challenge. The institutional setting differs from an independent research lab, but the principle transfers: a model must be evaluated as part of a decision process, not only as code that executes.

Measured in the lab#

Measured evidence. The current research program contains 204 theory-anchored quantitative tests and 16 publishing regression checks—220 automated checks in total. The distinction matters. The 204 quantitative checks evaluate identities, limiting cases, convergence, calibration, data behavior, and look-ahead protections. The 16 publishing checks protect public information architecture, metadata, links, identity, and evidence boundaries. Publishing checks do not become extra quantitative validation by being added to the total.

Six executed notebooks carry investigations across pricing, portfolio construction, derivatives, simulation, factor exposure, and market risk. The data layer uses a frozen 15-ETF price universe, recorded snapshots, hashes, declared seeds where randomness enters, and version-pinned source artifacts. The public research log records each question, claim, assumptions, dataset, method, falsification criterion, result, uncertainty, failure modes, and reproduction path.

The record also preserves unfavorable outcomes. A portfolio-construction claim was narrowed from superior performance to a more stable realized risk shape. A historical value-at-risk model was rejected after its breach frequency and clustering failed calibration tests. Rejection and revision are not defects in the record; they are evidence that the failure criteria can change the conclusion.

Established context. Formal model-risk guidance likewise treats development, validation, governance, documentation, and use as a connected system. A model can be implemented correctly and still be used outside the conditions its evidence supports.

Interpretive implication. The strongest signal of credibility is not that every test passes. It is that the process makes failure legible, preserves the rejected result, and narrows the public claim accordingly.

Five stages from question to conclusion#

A compact protocol turns the principle into practice.

1. Frame a falsifiable question. Name the decision, state the claim, and define what evidence would count against it before inspecting the final comparison.
2. Engineer the evidence. Record source, dates, transformations, missingness, version, and provenance. Treat a refreshed dataset as a new experiment.
3. Implement against anchors. Test code against theoretical identities, limiting cases, convergence behavior, and independent numerical methods—not only stored expected values.
4. Test the model. Use chronological holdouts, calibration tests, uncertainty intervals, stress scenarios, and multiplicity controls appropriate to the claim.
5. Publish the boundary. Report the result, uncertainty, failures, data vintage, source artifact, and the decision that changes. Keep rejected and revised claims discoverable.

The sequence is intentionally asymmetric. It is easy to produce another result; it is harder to specify in advance what would make the result untrustworthy.

What the evidence does not establish#

• Reproducibility shows that a procedure can be repeated; it does not show that its assumptions describe the market permanently.
• A passing test supports the behavior examined by that test. It does not certify every implementation path, dataset, instrument, or regime.
• A holdout reduces one form of overfitting. One holdout period does not represent every future environment.
• A confidence interval conditions on a sampling and model design. It cannot price uncertainty omitted from that design.
• Automated tests can verify declared properties. They do not replace independent review, peer review, economic judgment, or monitoring after use.
• Transparent code makes scrutiny possible. Openness alone does not make a conclusion correct.

The proper output is bounded confidence: enough evidence to support a scoped decision, paired with a clear account of what would require revalidation.

Review the protocol#

Review the lab's validation protocol. It provides the complete five-stage method and reusable checklists for model validation, portfolio stress testing, and reproducible research.