ADR-0003 — Deterministic aggregation; the LLM explains rather than scores¶
Status: Accepted · Date: 2026-08-03
Context¶
The natural design for a multi-agent investment system is to have specialist LLM agents analyse a candidate, then have a supervisor LLM read their outputs and produce a recommendation with a confidence score. It is intuitive, easy to build, and demos well.
Decision¶
Split the responsibilities. Analyst agents emit structured, evidence-cited signals. A deterministic gradient-boosted model, trained walk-forward and isotonically calibrated, produces the score and confidence. The Supervisor LLM writes the explanation and may veto from a closed set of reasons — but the score and confidence fields are not in its output schema, so it cannot alter them.
Rationale¶
LLM confidence is not a probability. A model asked for "confidence: 0.82" produces a token sequence, not a calibrated estimate. It cannot be backtested, it does not correspond to observed frequency, and telling a user "82% confident" on that basis is a fabrication. Since calibration is the system's primary trust claim (00 §6), a number that cannot be calibrated is disqualifying.
Free-form synthesis is unauditable. There is no way to attribute a natural-language judgement to specific inputs, which forecloses attribution analysis, agent-level evaluation, and the contribution bars that make the "Why?" screen honest. SHAP values over a tree model give exactly that, directly.
Twelve LLM calls are not twelve opinions. Calls to the same base model over overlapping context are heavily correlated. Averaging them produces the appearance of an ensemble with none of the variance reduction, at twelve times the cost.
Small prompt changes move outputs unpredictably, making it impossible to distinguish a genuine signal change from a prompt-edit artefact.
Meanwhile, LLMs are genuinely excellent at the tasks this design assigns them: reading unstructured filings and news, extracting structured claims with citations, and articulating reasoning in plain language. The architecture uses them where they are strong and refuses to use them where they are weak.
Consequences¶
Positive. Confidence means something and can be validated against outcomes. Every recommendation decomposes into per-agent contributions. Prompt changes are separable from model-behaviour changes. Cheaper — one tier-3 call instead of a synthesis debate.
Negative. Cold start: with no outcome history there is nothing to train, so the system opens with documented fixed weights, confidence capped at 0.65, and every output labelled pre-calibration. This is a real product cost, accepted because the alternative is a confidence number that lies from day one.
The aggregator also cannot capture reasoning an analyst expressed only in prose — which is why the Supervisor retains a categorised veto. That veto is the escape hatch for genuine qualitative judgement, bounded so it cannot become an uncalibrated model silently overriding a calibrated one, and every use is logged and reviewed.
Rejected alternative. LLM-as-judge synthesis with post-hoc calibration of its stated confidence. Rejected because the underlying signal is unstable across prompt and model versions, so the calibration would need refitting on every change and would never accumulate enough samples to be meaningful.