Monitoring democratic institutions through public records
Display:
← Back to overview
System Architecture
Tech Stack#
- Framework — Next.js 14 with Pages Router, TypeScript strict mode
- Styling — Tailwind CSS with custom design tokens (dm-* color system)
- Database — PostgreSQL with Drizzle ORM and pgvector extension for embedding storage and similarity search
- Caching — Redis with automatic in-memory fallback when Redis is unavailable
- AI Providers — OpenAI (GPT-4o-mini for Layer 2 Pass 1, text-embedding for Layer 3) and Anthropic (Claude Sonnet for Layer 2 Pass 2 and narratives)
- Testing — Vitest with jsdom environment
- Linting — ESLint (next/core-web-vitals), OpenGrep custom rules, Knip for dead code detection
- Package Manager — pnpm
Data Flow#
Documents move through the system in a defined pipeline:
Sources→Fetch→Store→Score→Aggregate→Embed→Assess→Narrate
- Fetch — Backfill pipeline fetches historical data; the weekly snapshot pipeline fetches incremental updates from all source types
- Store — Raw documents persisted to the
documentstable with source metadata - Score — Keyword annotation layer matches documents against category-specific dictionaries (capture/drift/warning tiers)
- Aggregate — Weekly rollups computed per category for structural analysis (volume, type distribution, agency counts)
- Embed — OpenAI text-embedding-3-small generates vector embeddings stored via pgvector for thematic drift analysis
- Assess — AI document review (sole active detection layer) drives concern status; structural anomaly, silence detection, and thematic drift are computed as descriptive context per category
- Narrate — AI generates expert and public narrative summaries for elevated categories
Source Types#
| Source | What It Provides | Fetcher Module |
|---|---|---|
| Federal Register | Executive orders, rules, notices, presidential documents | federal-register-fetcher.ts |
| CourtListener | Federal court opinions (stored as documents) and case docket metadata via the RECAP archive (feeds the tracked_cases litigation tracker) | courtlistener-fetcher.ts |
| DOJ Press Releases | Department of Justice press releases across divisions | doj-fetcher.ts |
| GovInfo | Congressional reports, public laws, presidential documents (CPD) | govinfo-fetcher.ts |
| FEC | Advisory opinions and Matters Under Review (MURs) | fec-fetcher.ts |
| CREC | Congressional Record floor speeches (Senate + House) | crec-fetcher.ts |
| CHRG | Congressional hearing transcripts (7 committees, subject-routed) | chrg-fetcher.ts |
| LegiScan | Federal legislative bill tracking via bulk datasets | legiscan-fetcher.ts |
| OIG | Inspector General reports from 11 offices: HHS, DOJ, SSA, DHS directly; OPM, TIGTA, Treasury, State, EAC, FEC, and the Intelligence Community via oversight.gov | hhs/doj/ssa/dhs-oig-fetcher.ts, oversight-gov-fetcher.ts |
| DHS Press | Operational press releases from DHS headquarters, ICE (full newsroom), and CBP (national media releases); HSI criminal-investigation releases also route to Federal Law Enforcement | dhs-press-fetcher.ts |
All source fetchers follow a consistent module pattern: parseParams() parses signal URLs, toContentItem() normalizes responses, fetchRecent() handles weekly snapshots, and fetchHistorical() handles backfill.
Database Schema#
Key tables in the PostgreSQL database:
| Table | Purpose |
|---|---|
| documents | Raw documents from all sources with title, content, publication date, source metadata |
| document_scores | Keyword annotation scores per document (tier, matched keywords, category) |
| weekly_aggregates | Per-category weekly rollups (volume, type distribution, agency counts, tier breakdown) plus the AI-driven concern status and descriptive layer scores |
| ai_document_assessments | Layer 2 AI assessment results (pass 1 flag, pass 2 classification, reasoning) |
| baselines | Historical baseline statistics per category (mean, stddev, distributions) |
| tracked_cases | Federal litigation case tracker: one row per case with court, filing/termination dates, status, posture, and category routing (refreshed weekly for open cases) |
| narratives | AI-generated expert and public narrative summaries |
Migrations are managed via Drizzle ORM with a schema-first workflow. The schema is defined in lib/db/schema.ts and migrations are generated into the drizzle/ directory.
Cron Jobs#
| Job | Schedule | What It Does |
|---|---|---|
| LegiScan Fetch | Monday 01:00 UTC | Downloads bulk legislative datasets from LegiScan, classifies bills into categories |
| Weekly Snapshot | Monday 03:00 UTC | Fetches new documents from all sources, runs AI assessment, computes aggregates, generates narratives |
| Database Dump | Monday 05:00 UTC | Creates a pg_dump backup on the persistent disk, served at /api/data/dump (GitHub Releases holds only the bootstrap dump) |
The snapshot pipeline orchestrates the full data flow: fetch, score, aggregate, embed, assess, and narrate. It is defined in lib/cron/snapshot.ts.
API Routes#
Server-side API routes act as proxies with caching and provide data endpoints for the dashboard:
/api/proxy— CORS proxy with host whitelist, content-type detection, Redis caching/api/federal-register— Federal Register API proxy/api/methodology— JSON endpoint exposing scoring thresholds and configuration constants
Deployment#
The application is configured for Render.com deployment via render.yaml:
- Web Service — Next.js application serving the dashboard and API routes
- PostgreSQL — Managed database with pgvector extension for embedding storage
- Redis — Key-value store for API response caching (600s default TTL)
- Cron Jobs — Three scheduled tasks (LegiScan fetch, weekly snapshot, database dump) running on Render's cron infrastructure