Skip to content

ADR-0006 — Local-first inference with gateway-enforced redaction

Status: Accepted · Date: 2026-08-03

Context

The system needs language models for extraction, analysis, and explanation. Cloud models are meaningfully stronger at the synthesis and explanation tasks that most affect user-visible quality. But the data involved — holdings, net worth, risk tolerance, and the user's religious practice — is exactly the data the product promises not to disclose.

The two goals conflict, and the usual resolutions are both bad: send everything to the cloud and put a privacy policy in settings, or refuse cloud entirely and ship a noticeably worse explanation.

Decision

Route by task tier, default to local, and enforce data minimisation at a single chokepoint.

  • All model access goes through one Model Gateway. No component calls a provider directly.
  • The gateway strips or refuses Private and Secret class fields on any outbound external call unless the corresponding user setting is enabled (10 §3).
  • Routing is configuration: tier 1 tasks (classification, extraction) go local always; tier 2 and 3 follow the profile, defaulting to local.
  • Every capability has a local fallback path. Nothing hard-fails on a missing API key.
  • The model actually used is recorded on every recommendation and shown when it differs from the preferred route.

Rationale

One chokepoint, not forty. A redaction rule applied at every call site is a rule that will be missed at the one added next month. Concentrating it in the gateway makes it testable: a single test asserts that no Private field can reach an external provider under default settings, run against every prompt template.

Task tiering makes the trade-off cheap. The overwhelming majority of calls are classification and extraction, where a local 8B model is entirely adequate and the volume would make cloud inference the system's dominant cost. Reserving cloud for tier-3 synthesis means the privacy exposure is limited to a small number of calls, on the smallest possible payload, only when the user has opted in.

Disclosure over silent degradation. When the system falls back to a local model for an explanation, it says so. A user who sees "explained by a local model" can judge the output accordingly; one who is not told cannot.

Consequences

Positive. The product works fully offline, which is a genuine differentiator rather than a checkbox. Marginal inference cost approaches zero. The privacy claim is verifiable by the user through the data-flow view, not merely asserted.

Negative. Local tier-3 synthesis on consumer hardware is slower and produces less fluent explanations than a frontier model, and explanation quality is the product. Users on the solo profile get a real quality reduction, disclosed rather than hidden. Supporting both paths also means every prompt must be tested against both local and cloud models, roughly doubling eval surface.

Rejected alternative. Cloud-only with a strong privacy policy. Rejected because it makes the central product promise contingent on a third party's behaviour, and because it would make the system unusable for a user who simply will not send their holdings anywhere.