03 — Data Platform¶

Everything the system claims must trace to a stored fact with a source and an as-of time. The data platform exists to make that true by construction.

1. Source catalogue¶

Sources are grouped by the guarantees they offer, because that determines how much we trust the derived signal.

Tier A — authoritative, primary, free¶

Source	Provides	Cadence	Notes
SEC EDGAR (XBRL + full-text)	US filings: 10-K, 10-Q, 8-K, DEF 14A, Forms 3/4/5, 13F	Real-time feed	Official. Structured XBRL gives normalised financials with exact filing timestamps — the gold standard for point-in-time fundamentals
FRED (St. Louis Fed)	~800k macro series: rates, CPI, GDP, employment, spreads	Per-series	Includes vintage/ALFRED data, i.e. what a series said on a past date before revision. Essential for honest macro backtests
Central bank publications	Policy decisions, statements, minutes	Scheduled	Calendar known in advance; used for event windows
Exchange corporate action notices	Splits, dividends, mergers, delistings	Daily	Correctness here determines whether historical prices are usable at all

Tier B — market and reference data¶

Source	Provides	Notes
Broker/data APIs (Alpaca, Polygon, Tiingo, EODHD)	OHLCV bars, quotes, corporate actions, symbol reference	Adapter interface; provider is swappable. Free tiers cover the laptop profile
Stooq / public end-of-day sources	Daily bars, indices, commodities, FX	Free fallback; lower reliability, used with staleness checks
Exchange calendars	Trading sessions, holidays, early closes	`exchange_calendars` library; wrong calendars silently corrupt every time-based feature

Tier C — narrative and sentiment (treated as untrusted input)¶

Source	Provides	Notes
GDELT	Global news event stream, tone scoring	Free, enormous coverage, noisy
Publisher RSS / official newsroom feeds	Financial and company news	Respect robots.txt and ToS; feeds only, no scraping of restricted content
Reddit API, StockTwits	Retail sentiment, volume-of-mention	Official APIs with rate limits; heavy bot contamination requires filtering
Earnings call transcripts	Management commentary, Q&A	Where licensing permits; otherwise derived from 8-K exhibits

Everything in Tier C enters the quarantine boundary (10 §4). It is attacker-controlled text and is never permitted near a tool-calling context.

Explicitly not used¶

Paywalled content accessed by circumvention; scraped data from sources whose terms prohibit it; any dataset whose redistribution terms we cannot satisfy. This constrains coverage and that is accepted.

2. Ingestion architecture¶

graph LR
    subgraph Acquire
        SCH["Dagster schedules<br/>+ streaming consumers"]
        ADP["Source adapters<br/>rate-limited, retrying"]
    end
    subgraph Land
        RAW[("Raw zone<br/>immutable, object store")]
    end
    subgraph Normalise
        VAL["Validation<br/>Pandera / Great Expectations"]
        NRM["Normalisation<br/>to canonical schema"]
        REC["Reconciliation<br/>cross-source"]
    end
    subgraph Serve
        TS[("Time series<br/>TimescaleDB")]
        FND[("Fundamentals<br/>bitemporal")]
        DOC[("Documents<br/>+ Qdrant chunks")]
        FEAT[("Feature store")]
    end
    SCH --> ADP --> RAW --> VAL --> NRM --> REC
    REC --> TS & FND & DOC
    TS & FND & DOC --> FEAT

Raw zone is immutable. Every fetched payload is stored verbatim with its request metadata and a content hash before anything parses it. When a parser bug is found six months later — and it will be — the fix is a reprocess, not a re-fetch of data that may no longer be available in its original form. This also makes citations resolvable to the exact bytes that were read.

Validation before normalisation. Schema, range, and cross-field checks run as a gate. A price series with a 90% single-day gap and no corresponding corporate action is quarantined for review, not written. Silent bad data is worse than missing data, because missing data is visible.

Reconciliation across sources. Where two sources cover the same fact (a closing price, a share count), disagreement beyond tolerance raises a data-quality event and marks the fact disputed. Agents treat disputed facts as reduced-confidence inputs.

Idempotency. Every write is keyed (source, entity_id, event_time, source_version). Replaying a day is safe and is the standard recovery procedure.

3. Canonical data model¶

Simplified, with the columns that carry the design intent.

-- Instrument reference, slowly changing
CREATE TABLE instrument (
    id                UUID PRIMARY KEY,
    figi              TEXT,              -- vendor-neutral identity; tickers get reused
    isin              TEXT,
    primary_ticker    TEXT NOT NULL,
    exchange_mic      TEXT NOT NULL,
    name              TEXT NOT NULL,
    sector_gics       TEXT,
    country_domicile  TEXT,
    currency          TEXT NOT NULL,
    listed_from       DATE NOT NULL,
    listed_to         DATE,              -- NULL = active. NEVER delete a delisted row.
    UNIQUE (figi)
);

-- Price bars (Timescale hypertable, partitioned on ts)
CREATE TABLE bar_daily (
    instrument_id UUID NOT NULL REFERENCES instrument(id),
    ts            TIMESTAMPTZ NOT NULL,     -- session close, exchange tz normalised to UTC
    open  NUMERIC(20,6) NOT NULL,
    high  NUMERIC(20,6) NOT NULL,
    low   NUMERIC(20,6) NOT NULL,
    close NUMERIC(20,6) NOT NULL,
    volume        BIGINT NOT NULL,
    adj_factor    NUMERIC(20,10) NOT NULL DEFAULT 1,  -- cumulative split/dividend factor
    source        TEXT NOT NULL,
    available_at  TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (instrument_id, ts, source)
);

-- Fundamentals: bitemporal. This table is where look-ahead bias lives or dies.
CREATE TABLE fundamental_fact (
    instrument_id  UUID NOT NULL REFERENCES instrument(id),
    metric         TEXT NOT NULL,            -- 'total_debt', 'revenue', 'cash_and_equivalents'
    period_end     DATE NOT NULL,            -- fiscal period the value describes
    period_type    TEXT NOT NULL,            -- 'Q' | 'FY' | 'TTM'
    value          NUMERIC(28,6),
    unit           TEXT NOT NULL,
    filed_at       TIMESTAMPTZ NOT NULL,     -- when the filing was made public
    available_at   TIMESTAMPTZ NOT NULL,     -- when WE could have known it
    restatement_of UUID,                     -- points at the superseded fact
    source_doc_id  UUID NOT NULL REFERENCES source_document(id),
    PRIMARY KEY (instrument_id, metric, period_end, period_type, filed_at)
);

-- Every document that any claim can cite
CREATE TABLE source_document (
    id            UUID PRIMARY KEY,
    source        TEXT NOT NULL,
    doc_type      TEXT NOT NULL,             -- 'filing_10k' | 'news' | 'transcript' | 'macro_release'
    title         TEXT,
    url           TEXT,
    published_at  TIMESTAMPTZ NOT NULL,
    available_at  TIMESTAMPTZ NOT NULL,
    content_hash  TEXT NOT NULL,
    raw_uri       TEXT NOT NULL,             -- object store location of the verbatim bytes
    trust_tier    TEXT NOT NULL              -- 'A' | 'B' | 'C'  → drives quarantine handling
);

-- Corporate actions, applied to produce adj_factor
CREATE TABLE corporate_action (
    id             UUID PRIMARY KEY,
    instrument_id  UUID NOT NULL REFERENCES instrument(id),
    action_type    TEXT NOT NULL,            -- 'split' | 'cash_dividend' | 'stock_dividend'
                                             -- | 'merger' | 'spinoff' | 'delisting' | 'symbol_change'
    ex_date        DATE NOT NULL,
    record_date    DATE,
    pay_date       DATE,
    ratio          NUMERIC(20,10),           -- splits
    amount         NUMERIC(20,6),            -- dividends, in `currency`
    currency       TEXT,
    announced_at   TIMESTAMPTZ NOT NULL,
    available_at   TIMESTAMPTZ NOT NULL,
    source_doc_id  UUID REFERENCES source_document(id)
);

Three things in this schema are doing disproportionate work:

instrument.listed_to instead of deletion. Delisted and bankrupt companies stay in the database forever. Deleting them is precisely how survivorship bias enters a backtest, and it is invisible once done.
fundamental_fact keyed by filed_at, with restatement_of. Companies restate. A backtest that uses today's restated figure for a decision made three years ago is testing a strategy that could not have been run. Keeping every vintage makes the correct query the natural one.
available_at on everything. The universal filter. See below.

4. Point-in-time correctness¶

The single most important invariant in the system:

No computation may read a fact whose available_at is later than the decision timestamp.

This is enforced at three levels.

API level. The feature store's only read interface takes an as-of timestamp. There is no function that returns "the current value of X" without one; the live path simply passes now(). A developer cannot accidentally write look-ahead code because the non-look-ahead call is the only call.

class FeatureStore(Protocol):
    def get(
        self,
        instrument_id: InstrumentId,
        features: Sequence[FeatureName],
        as_of: datetime,          # mandatory, no default
    ) -> FeatureVector: ...

Database level. Every serving view carries WHERE available_at <= :as_of, and raw table access outside the feature store is restricted by grant.

Test level. A dedicated audit harness replays historical decisions with a database snapshot artificially truncated at the decision time, and asserts identical output. Any divergence means something read the future. This runs in CI over a fixed golden set.

available_at is not always the same as published_at, and the difference matters. A 10-K filed at 16:31 ET is public immediately, so available_at = filed_at. A macro series revised silently gets available_at at the revision publication, not the original period. A vendor that delivers data on a T+1 batch gets available_at at delivery, not at the event. Getting these right per source is a documented per-adapter responsibility.

5. Feature store¶

Features are versioned, documented, point-in-time computed transformations. Each has an owner module, a definition, a dependency list, and a staleness tolerance.

- name: rsi_14
  category: technical
  dtype: float
  range: [0, 100]
  depends_on: [bar_daily.close]
  lookback: 15 sessions
  staleness_tolerance: 1 session
  null_policy: emit_null    # never forward-fill a technical indicator

- name: debt_to_market_cap
  category: shariah_screen
  dtype: decimal
  depends_on:
    - fundamental_fact.interest_bearing_debt
    - market_cap_trailing_avg_24m
  staleness_tolerance: 100 days      # a quarter plus filing lag
  null_policy: fail_closed           # missing input ⇒ UNCERTAIN verdict, never a pass

- name: earnings_surprise_pct
  category: fundamental
  depends_on: [fundamental_fact.eps_actual, consensus.eps_estimate]
  staleness_tolerance: 1 quarter
  null_policy: emit_null

null_policy is the interesting field. Forward-filling a stale value is the most common way a data platform lies to a model. Technical indicators emit null and the consuming agent abstains. Compliance ratios fail closed — a missing debt figure produces UNCERTAIN, which excludes the candidate. Failing closed on compliance and failing open nowhere is the rule.

Feature families:

Family	Examples
Technical	RSI, MACD, ATR, realised vol, Bollinger position, relative strength vs index, volume z-score, 52-week position, trend regime
Fundamental	Margins, ROIC, ROE, FCF yield, revenue/earnings growth (multi-period), accruals, leverage, interest coverage, dilution, valuation multiples
Quality & risk	Piotroski F-score, Altman Z, earnings variability, beta, downside beta, max drawdown, liquidity (ADV, spread)
Macro	Yield curve level/slope/curvature, real rates, CPI surprise, PMI, unemployment change, credit spreads, USD index, commodity levels
Event	Days to/from earnings, insider net buying (Form 4), institutional ownership change (13F, with its 45-day lag honoured), analyst revision breadth
Narrative	News volume z-score, sentiment (model-scored, source-weighted), topic tags, social mention velocity, bot-filtered sentiment
Compliance	Every ratio input and output in 05

Computation: batch nightly via Dagster partitioned assets; on-demand for ad-hoc as-of queries; Redis cache keyed on (instrument, feature, as_of_bucket, feature_version).

6. Document corpus and retrieval¶

Every document that can be cited is chunked, embedded, and indexed with metadata: instrument_ids, doc_type, published_at, available_at, trust_tier, section (for filings — Item 1A Risk Factors is a different retrieval target than Item 7 MD&A).

Retrieval is hybrid — BM25 for exact terms like ticker symbols, metric names, and legal phrasing, plus dense vectors for semantics — fused with reciprocal rank fusion, then reranked by a small cross-encoder.

Two constraints on retrieval that are unusual and load-bearing:

Retrieval is time-filtered. Every query carries available_at <= as_of. RAG is a look-ahead vector too: retrieving next quarter's earnings release when explaining a decision made last quarter produces a confident, wrong, and convincing explanation.
Retrieval results carry trust tier through to generation. A claim sourced from a Tier C blog post is rendered differently in the UI than one from an SEC filing, and the aggregator weights them differently.

7. Licensing and terms of service¶

This section is a design constraint, not boilerplate.

Every adapter records the licence and redistribution terms of its source in code, next to the adapter.
Data whose terms forbid redistribution is stored locally and never leaves the deployment. Since this is a single-user private system, that is compatible with most retail data licences — but it hard-blocks the multi-tenant path, which is one more reason multi-tenancy is out of scope.
Rate limits are respected by construction: adapters have declared quotas and a token-bucket limiter; exceeding a quota fails the job rather than hammering the provider.
robots.txt and ToS are honoured. Where a source prohibits automated access, we do not use it, and the coverage gap is documented rather than quietly worked around.
Personal data appearing incidentally in filings (executive names in Form 4s, for instance) is retained only as needed and never used for anything but the disclosed insider-transaction feature.

8. Data quality monitoring¶

Continuous checks, surfaced in-app rather than only in logs, because data quality directly bounds how much the user should trust a recommendation:

Freshness per source and per feature family, against declared tolerance.
Completeness — coverage ratio of the universe per feature; a sudden drop signals an upstream schema change.
Distribution drift — population stability index on feature distributions week over week; catches unit changes and silent provider migrations.
Cross-source agreement — disagreement rate on overlapping facts.
Corporate action integrity — any price gap beyond a threshold must have a matching corporate action, or it is flagged.
Point-in-time audit — the CI replay described in §4.

The recommendation card in the app shows a data-quality badge derived from these. A recommendation built on stale or disputed inputs says so on its face.