Traceable data
Sources, dates, transformations, missingness, hashes, and frozen vintages remain attached to the result.
Research design and validation
Research design, model validation, and reproducibility
The test comes before the conclusion. Every investigation declares its question, data, uncertainty, and failure criteria first.
Validation standard
Validation compares a model with pre-specified theoretical and empirical criteria; every conclusion remains conditional on the test design.
Purpose
Determine whether a result is reliable enough to inform a scoped allocation, risk, pricing, or governance decision.
Evidence required
Theoretical anchors, untouched data, uncertainty estimates, failure criteria, versioned artifacts, and reproducible execution.
Result classification
A validated, supported, rejected, or revised claim. The reason and the affected decision are stated.
Boundary
Passing validation does not make the model the market. It does not promise that future data will resemble the test.
Validation protocol
Five stages from research design to publication
- 01
Frame a falsifiable question
Name the decision and state the claim. Define in advance what evidence would count against it. “Explore the data” is not a falsification criterion.
- 02
Engineer the evidence
Document selection, dates, transformations, missingness, hashes, and provenance. Separate refreshing data from reproducing an existing experiment.
- 03
Implement against anchors
Test numerical code against identities, limiting cases, convergence behavior, and independent methods. A stored fixture is not an anchor.
- 04
Test the model
Use chronological holdouts, calibration tests, stress scenarios, uncertainty intervals, and multiplicity controls appropriate to the claim.
- 05
Publish the boundary
Report the result, uncertainty, failure modes, data vintage, source commit, and the decision that changes. Rejections and revisions stay published.
Evidence standards
Three conditions for an inspectable result
Testable models
Theoretical identities, limiting cases, convergence behavior, holdouts, calibration, and stress tests define what can fail.
Bounded conclusions
Uncertainty, limitations, rejected claims, revisions, and the affected decision are reported together.
Evidence classification
Six classes prevent context from masquerading as proof
Lab measurement
A value generated from declared data, transformations, estimators, and uncertainty procedures.
Numerical validation
Agreement with an identity, limiting case, convergence rate, or independently implemented method.
External empirical evidence
An identified study or dataset evaluated separately from the lab's own measurement.
Institutional record
A dated decision, release, legal instrument, or official account. It establishes chronology and stated action, not causality.
Interpretive synthesis
A bounded connection among measured results, external records, mechanisms, and decision consequences.
Proposed research
A question, baseline, competing explanation, and publication gate declared in advance. No supported result yet.
Reusable resources
Three checklists for analytical work
Model-validation checklist
Use before approving a model for decision support.
- Name the decision and the model owner.
- State assumptions explicitly and make each one testable.
- Anchor the model to theory or an independent method.
- Prevent leakage by construction, not by inspection.
- Test calibration and residual behavior.
- Attach uncertainty to every reported result.
- Include stress conditions and boundary cases.
- Declare failure criteria before the comparison runs.
- Write limitations in decision language.
- Define monitoring and the revalidation trigger.
Portfolio stress-test checklist
Use before accepting an allocation because it looks diversified.
- Stress expected returns and estimation error separately.
- Challenge correlation, not only asset volatility.
- Measure concentration by weights and risk contribution.
- Include liquidity, turnover, and transaction costs.
- Test chronological out-of-sample behavior.
- Replay dated crises without treating them as bounds.
- Map activity, inflation, liquidity, and institutional response separately.
- Compare simple baselines to optimized portfolios.
- Report drawdown and recovery, not only volatility.
- State which risks remain outside the scenario set.
Research-reproducibility checklist
Use before calling an analytical result reproducible.
- Pin code to a commit or release.
- Record data source, vintage, transformations, and hash.
- Separate data refresh from experiment reproduction.
- Declare seeds and random-generator family.
- Document runtime and dependency requirements.
- Preserve executed artifacts and machine-readable outputs.
- Test a clean-clone reproduction path.
- Publish uncertainty and numerical error budgets.
- Link prose claims to the generating artifact.
- Keep rejected and revised claims discoverable.
Inspect the record
Inspect implementation and evidence
Full audit trail
The research record preserves how each conclusion changed.
Questions, rejected claims, revisions, data versions, source commits, notebooks, and known limitations remain available for inspection.