08 — Backtesting¶
1. What a backtest is for¶
A backtest cannot tell you a strategy will work. It can only tell you a strategy would not have worked, which is still the most valuable thing available before risking money. The engine is therefore designed around one goal: making it hard to produce a flattering result by accident.
Most retail backtesting tools optimise for the opposite — easy to run, easy to look good, silent about the assumptions that produced the number. Every design choice below trades convenience for the ability to believe the output.
2. Architecture¶
The backtest engine runs the same strategy, compliance, risk, and fill code as the live paper path. Only the clock and the data source differ.
graph LR
CFG["Run manifest<br/>strategy, universe,<br/>period, costs, seed"] --> ENG
PIT[("Point-in-time<br/>feature store")] --> ENG
ENG["Event loop<br/>advances a virtual clock"]
ENG --> STR["Strategy / Agent path<br/>(identical to live)"]
STR --> SHR["Shariah gate<br/>(historical verdict)"]
SHR --> RSK["Risk engine<br/>(identical to live)"]
RSK --> FIL["Fill engine<br/>(identical to live)"]
FIL --> LED["Ledger"]
LED --> MET["Metrics + artifacts"]
MET --> REP["Report + reproducibility hash"]This shared-code property is not a convenience; it is the correctness argument. A backtester with its own parallel implementation of position sizing tests that implementation, not the system.
The virtual clock is the only source of "now." No component may call datetime.now(). A lint rule forbids it outside the clock module, and a test asserts that a backtest run produces identical results when the wall clock is shifted. Time leaking in through a system call is the single most common source of look-ahead in real backtesting code.
3. Biases the engine structurally prevents¶
Each of these is a way to make a bad strategy look good. The design makes each one require deliberate effort rather than mere carelessness.
Look-ahead bias¶
Every read goes through FeatureStore.get(..., as_of=clock.now()) with a mandatory as-of parameter (03 §4). Fundamentals are keyed by filed_at, so the engine sees a restated figure only after the restatement was filed. Retrieval for RAG carries the same time filter. Signals fill on subsequent bars (06 §4).
Survivorship bias¶
The universe is reconstructed as of each historical date from instrument.listed_from/listed_to — including companies that later went bankrupt, were acquired, or delisted. Delisted instruments are never deleted. A backtest run on today's index membership silently excludes every failure and can add several points of annual return out of nothing.
Compliance look-ahead¶
The Shariah gate uses the verdict as it stood on the historical date, recomputed from point-in-time fundamentals and that date's rule pack version. Screening a historical strategy with today's compliance list is a subtle and substantial look-ahead, because compliance status correlates with leverage and leverage correlates with distress.
Data-snooping and multiple testing¶
Every backtest run is logged, per strategy family, with its parameters. The report shows the number of configurations tried and reports the Deflated Sharpe Ratio (Bailey & López de Prado), which adjusts significance for the number of trials and for return skew and kurtosis. Testing 200 variants and reporting the best is guaranteed to find something; the report makes that visible instead of letting it be forgotten.
Overfitting to a single period¶
Walk-forward is the default and in-sample-only results are labelled as non-evidence. See §4.
Cost optimism¶
Costs are on by default and cannot be set to zero without the report carrying a prominent UNREALISTIC_COSTS flag. Fill fidelity level is printed on every report (06 §8).
Corporate action errors¶
Prices are adjusted from the corporate action table, and gaps without a matching action are flagged by the data quality monitor (03 §8). Unadjusted splits create fictional 50% single-day losses; incorrectly adjusted dividends quietly inflate returns.
4. Validation methodology¶
Walk-forward analysis (default)¶
Anchored or rolling. Only the concatenated out-of-sample segments are reported as performance. In-sample results appear in the report solely as a diagnostic — a large in-sample/out-of-sample gap is itself the finding.
Purged K-fold with embargo¶
For the aggregator and any other model trained on overlapping-horizon labels. Standard K-fold leaks badly with financial data: a label spanning 20 forward days overlaps training observations, so information crosses the fold boundary. Following López de Prado, training observations whose label windows overlap the test set are purged, and an additional embargo period after the test fold is excluded to handle serial correlation.
Combinatorial purged cross-validation¶
Generates multiple backtest paths rather than one, yielding a distribution of outcomes instead of a single number. The right question is not "what was the Sharpe?" but "across plausible paths, how often was this strategy acceptable, and how bad was the worst path?"
Regime segmentation¶
Every result is broken down by market regime — bull, bear, high-vol, low-vol, rising rates, falling rates. A strategy that made all its money in one regime has a regime bet, not an edge, and should be sized accordingly.
Monte Carlo robustness¶
Bootstrap resampling of trade sequences to produce a drawdown distribution; parameter perturbation to confirm results are not on a knife-edge; randomised entry timing (±1–2 days) to check the edge does not depend on exact-day precision that will not survive real execution.
5. Metrics reported¶
| Group | Metrics |
|---|---|
| Return | CAGR, cumulative, monthly/annual table, best/worst periods |
| Risk | Volatility, downside deviation, max drawdown, drawdown duration and recovery time, Ulcer index, VaR/CVaR |
| Risk-adjusted | Sharpe (vs 0% baseline, per 06 §7), Sortino, Calmar, Deflated Sharpe, Probabilistic Sharpe |
| Benchmark-relative | Alpha, beta, tracking error, information ratio, up/down capture vs a compliant benchmark |
| Trade | Count, hit rate, avg win/loss, profit factor, expectancy, avg holding period, turnover, exposure |
| Cost | Total drag decomposed by commission, spread, impact, FX; gross vs net |
| Compliance | Universe compliance rate over time, positions blocked by the gate, purification accrued |
| Attribution | By sector, geography, agent, and signal type |
| Capacity | Estimated maximum deployable capital before impact degrades returns beyond a threshold |
Capacity is unusual to include and worth the effort: a strategy that works at £10k and dies at £500k is fine to know about in advance, and impossible to discover from a returns curve.
6. The report¶
Every run produces a reproducibility manifest:
run_id: bt_01J9X...
code_commit: a3f9c21
data_snapshot: 2026-08-03T00:00:00Z
rule_pack: aaoifi_ss21@2024.1
model_versions: {supervisor: claude-opus-5, analysts: qwen3-32b@q4}
prompt_hashes: {technical: 7f3a..., fundamental: 1c8e...}
aggregator_version: gbm-2026-06@iso-cal-v3
universe_def: compliant_us_adv>5m
period: [2019-01-01, 2026-06-30]
costs: {commission: per_share_0.005, fidelity: L2}
seed: 42
config_trials_in_family: 14 # feeds Deflated Sharpe
result_hash: 9d41ba...
Re-running from this manifest must reproduce result_hash exactly. This is enforced as a CI test on a golden strategy, which catches the class of change that silently alters historical results.
The report opens with the caveats, not the equity curve — fidelity level, number of trials, out-of-sample proportion, and any active warning flags — because those determine whether the curve below means anything.
7. In-app backtesting¶
The in-app Backtesting view is deliberately constrained. It offers strategy templates, a universe and period selector, and cost presets; it does not offer a free parameter sweep, because a phone-based optimiser is a machine for generating overfit strategies.
Results always display: out-of-sample equity curve with the in-sample portion visually distinct; drawdown chart; the compliant naive benchmark alongside; regime breakdown; the trials count and Deflated Sharpe; and a plain-language verdict such as "This result comes from 3 of 7 years out-of-sample and does not clear the naive compliant benchmark after costs."
Saying that plainly is the whole point. A backtesting feature that only ever produces encouraging results is worse than none.