Logan Jonesupdated jul 2026
← index

MarketLens

2026 · self-hosted · in daily use · github

Ask a chatbot about the market and it answers from parametric memory: unsourced, unreproducible, stale. MarketLens takes the opposite bet. Ingest the primary sources, embed and cluster everything, and compute research from the corpus. The local LLM gets three narrow jobs and no opinions.

fig. 1 · the week across assets: indexes, sectors, rates, movers · [zoom]

fig. 1 · the week across assets: indexes, sectors, rates, movers[click] zoom in · [esc]

The pipeline

Five stages, every one auditable back to rows in Postgres.

  • Ingestion: SEC EDGAR per-CIK, wire RSS, company IR feeds, FRED/BLS/BEA macro series, Fed communications, FTC/DOJ/SEC enforcement, the Federal Register, CourtListener litigation, Reddit, Finnhub. Earnings calls transcribed on-host by Whisper.
  • Embedding: every article gets a 1024-dim embedding, then pgvector nearest-neighbor assignment into deduplicating clusters, 7-day window, 0.85 cosine.
  • Triage: a zero-shot classifier gates each cluster against a typed event vocabulary. Below threshold, the cluster never reaches the LLM.
  • Extraction: the local model sees a cluster, never a single headline. Constrained JSON out: slots, sentiment, magnitude. The event type is given, not asked.
  • Importance: source weight × novelty × event-class prior × magnitude. Arithmetic. The model never produces the score.

edgar · fred · bls · bea · fed · ftc · doj · courtlistener · federal register · business wires · ir feeds · reddit · finnhub

The research workspace

Theses, not chat sessions. A thesis is a persistent hypothesis with matching rules and an accumulating evidence ledger. A matcher scans new articles, filings, and call transcripts every 60 seconds and binds hits with the reason and similarity that produced them. Stance classification runs against the thesis statement: supports, contradicts, neutral, with a one-line rationale. Conviction comes from the ledger, not from asking the model how it feels.

fig. 2 · thesis workspace: sub-tracks, evidence ledger, stance · [zoom]

fig. 2 · thesis workspace: sub-tracks, evidence ledger, stance[click] zoom in · [esc]

How it’s built

.NET 10 API, React frontend, Postgres with pgvector. Three Python sidecars: embeddings, zero-shot triage, Whisper transcription. Ollama on the host for the constrained-JSON jobs. Everything runs on one machine; no data leaves it.

A Postgres work queue with leases, retries, and dead-lettering drives the drains. A pipeline run ledger records every cycle: inputs, outputs, skips, errors by category. Green sources tell you ingestion ran; a green ledger tells you events actually landed.

The hard parts

  • Scoping the LLM. Every step that can be arithmetic is arithmetic; the model is reserved for judgments where it has a real edge, and each one is bounded by a thesis statement and grounded in cluster context. There is no “ask the model about the market” surface. That is the design.
  • Source trust. Five reputation tiers from EDGAR down to Reddit, with explicit weights. The math is supposed to drown low-trust breadth under low importance, and the tier system is what makes an aggregator headline lose to a filing.
[0] ~/portfoliominnesota