11 — Learning & Evaluation

1. The problem with "the AI learns from results"

Financial markets are the worst possible environment for naive learning loops. The signal-to-noise ratio is extreme, the data-generating process is non-stationary, feedback is delayed by weeks, sample sizes are small, and — uniquely — the learner's own actions can invalidate the pattern it learned. A system that retrains nightly on last week's P&L will confidently learn noise and then act on it with increasing size.

So this system's learning design is built around a constraint that most of its marketing-equivalent claims ignore:

Improvement must be slower than the temptation to improve. Learning happens offline, on out-of-sample evaluation, with regime awareness, behind explicit promotion gates, and it never touches a live decision path directly.

What genuinely improves, in descending order of reliability:

1. Calibration — how confidence maps to observed frequency. Large samples, fast feedback, low risk.
2. Agent weighting — which analysts deserve influence in which regimes. Moderate samples, moderate risk.
3. Retrieval memory — surfacing analogous past cases. No parameters to overfit.
4. Prompt and rubric refinement — driven by a human-reviewed failure taxonomy, not by gradient.
5. Feature engineering — the slowest and highest-value, essentially research.

What is deliberately not done: online reinforcement learning on live outcomes, automatic strategy parameter optimisation on recent performance, or any automated change that increases position size in response to recent wins. Those are the mechanisms by which systems blow up.

2. Outcome recording

Every recommendation is a labelled prediction the moment it is made. At issue time the system stores the full manifest (01 §5); at resolution it stores what happened.

class RecommendationOutcome(BaseModel):
    recommendation_id: UUID
    resolved_at: datetime
    resolution: Literal["TARGET_HIT", "STOP_HIT", "HORIZON_ELAPSED",
                        "THESIS_INVALIDATED", "COMPLIANCE_EXIT", "USER_EXIT"]

    realised_return: Decimal
    benchmark_return: Decimal            # same window, compliant benchmark
    excess_return: Decimal
    vol_adjusted_return: Decimal         # 3% in a quiet name ≠ 3% in a volatile one
    max_adverse_excursion: Decimal       # worst drawdown before resolution
    max_favourable_excursion: Decimal    # best unrealised gain — was the target too far?

    favourable: bool                     # the calibration label
    agent_signals: list[Signal]          # what each agent said, preserved
    agent_attribution: dict[AgentName, float]   # SHAP at decision time
    regime_at_entry: RegimeLabel
    user_action: Literal["TRADED", "SKIPPED", "MODIFIED"]

Three of these fields do disproportionate work. Volatility-adjusted return prevents the system from learning that volatile stocks are good signals. Max adverse excursion distinguishes a correct call the user could not have held from a comfortable one — a strategy that is right after a 30% intermediate drawdown is not a usable strategy. Skipped recommendations are tracked too, which is essential: only learning from taken trades produces selection bias, and the skips are also how user preference is inferred.

3. Calibration

The primary learning loop, and the one that most directly earns trust.

An isotonic regression maps the aggregator's raw score to empirical favourable-outcome frequency, refit on a rolling window with a minimum sample threshold, fitted separately per horizon bucket and per market regime — because a system well-calibrated in a bull market is typically overconfident in a bear one.

Tracked and displayed in-app:

• Brier score and its decomposition into reliability, resolution, and uncertainty. Reliability is calibration quality; resolution is whether the system discriminates at all. A system can be perfectly calibrated and useless (always predicting the base rate), so both matter.
• Reliability diagram — predicted vs observed, with confidence bands.
• Expected Calibration Error by bucket.

The target from 00 §6: the 70% bucket resolves favourably 65–75% of the time. Until enough samples exist, confidence is capped and every output is labelled pre-calibration. Claiming calibration you have not demonstrated is the specific dishonesty this section exists to prevent.

4. Agent weighting

The aggregator is retrained on a schedule (monthly by default), strictly walk-forward, with promotion gated on out-of-sample improvement across all of: overall Brier score, hit rate, and per-regime performance. A candidate that improves the average while degrading bear-market performance is not promoted — the average is the wrong objective when the tail is what hurts.

Per-agent diagnostics maintained continuously:

Metric	What it detects
Standalone hit rate by regime	An agent that only works in one environment
Calibration of self_confidence	Systematic over/under-confidence, correctable in the aggregator
Marginal contribution (leave-one-out)	An agent adding nothing beyond what others already say
Abstention rate and its accuracy	Whether abstentions are appropriately targeted
Correlation with other agents	Redundancy — two agents saying the same thing are one agent
Evidence quality	Citation validity rate, trust-tier mix

An agent whose marginal contribution is indistinguishable from zero across a full evaluation window is a candidate for removal. Keeping twelve agents because twelve is the design is how systems accumulate expensive noise; the roster is subject to the same evidence standard as everything else.

5. Case memory

Resolved recommendations are embedded and indexed, so a new candidate can retrieve analogous historical cases: similar setup, similar regime, similar evidence pattern — and what happened.

This is retrieval, not parameter learning, which is why it is safe: nothing is fitted, so nothing can overfit. It enters the pipeline as context to the Supervisor ("three similar setups in this regime resolved unfavourably, principally because…") and as a user-facing feature on the recommendation card. Retrieval is time-filtered like all other retrieval (03 §6).

6. Failure taxonomy

Every unfavourable outcome is categorised, by an automated first pass with human review of a sample:

Category	Meaning	Response
THESIS_WRONG	Reasoning was sound, the world differed	Usually nothing — this is the irreducible cost of forecasting
EVIDENCE_MISREAD	An agent misinterpreted a source	Prompt fix, eval case added
DATA_ERROR	Input was wrong or stale	Data platform fix, validation rule added
STALE_ASSUMPTION	A relationship stopped holding	Feature review, regime detection improvement
REGIME_MISMATCH	Correct in one environment, applied in another	Regime classification improvement
TIMING	Right idea, wrong entry	Execution planner tuning
SIZING	Correct call, position too large or small	Risk engine review
COMPLIANCE_DRIFT	Instrument's status changed mid-hold	Monitoring buffer tightening

Separating THESIS_WRONG from the rest is the discipline that matters most. Being wrong at an acceptable rate is the job; the categories that warrant change are the ones where the process failed, not the ones where the future was uncertain. Systems that treat every loss as a defect tune themselves into overfitting.

7. Evaluation harness

Runs in CI on every change to prompts, models, features, or rules — because a prompt edit is a behaviour change and should be treated with the same suspicion as a code change.

Agent-level evals. Golden sets per agent: instrument-and-date pairs with expert-labelled expected direction and required evidence citations. Scored on directional accuracy, citation validity (does the cited document actually support the claim), abstention appropriateness, and schema conformance.

Compliance regression. The full suite from 05 §10: constituency agreement, golden set, property tests, point-in-time replay.

End-to-end replay. A fixed set of historical dates replayed through the full pipeline against a truncated database, asserting reproducibility and no look-ahead.

Explanation faithfulness. Automated checks that every numeric claim in generated text matches the evidence set, that citations resolve, and that no claim appears without support. An LLM judge scores clarity; the factual checks are deterministic, because using a model to check a model's arithmetic compounds rather than catches errors.

Red-team suite. Prompt injection (10 §4), plus adversarial market conditions: halts, gaps, missing data, contradictory sources, extreme volatility.

Cost and latency budgets as regression tests, so quality improvements that quietly triple inference cost are visible.

8. Model and version management

Everything that affects output is versioned and recorded on every recommendation: prompt template hashes, model identifiers and quantisation, feature definition versions, aggregator and calibration versions, rule pack versions, and code commit.

Promotion is gated: a candidate version must pass the full eval suite, show non-degraded performance on the golden sets, and — for aggregator changes — demonstrate walk-forward improvement. Shadow mode runs the candidate alongside production without acting on it for a defined period first.

Rollback is a version pointer change, and because every recommendation records the versions that produced it, "when did behaviour change and why" is answerable rather than archaeological.

9. User feedback

Two channels, weighted very differently.

Explicit — thumbs on explanation quality, and a reason when a recommendation is skipped. This trains presentation and preference: explanation style, which factors the user cares about, tolerance for particular sectors or setups. It does not train the prediction model. A user's opinion at recommendation time contains no information about the future price, and letting it in would import their biases into the estimator while appearing to personalise.

Implicit — which recommendations are traded, held, or exited early, and how sizing is modified. Used for personalisation of surfacing and risk defaults, with the same firewall.

The separation is deliberate and worth restating: user feedback shapes what and how the system communicates; only market outcomes shape what it predicts.

10. Guardrails on learning

• No automated change increases position sizing or risk limits. Ever. Those move only by explicit user action.
• No retraining on windows shorter than a full market cycle for anything structural.
• Every promoted model change is announced in-app with what changed and why, because silent behaviour change is corrosive to a system built on trust.
• Performance degradation beyond a threshold triggers automatic rollback to the last known-good version and a circuit breaker (07 §6).
• The learning pipeline has no write access to production configuration. It produces candidate versions; promotion is a separate, gated, logged action.