Monitoring democratic institutions through public records
Display:
← Back to Data
Data Dictionary
Every field in the downloadable datasets: what it means, how it is derived, and what to watch out for. A machine-readable version is at /api/export/dictionary. This page is generated from the same registry the pipeline is tested against — a guard test fails the build if it drifts from the actual schema.
Weekly Aggregates CSV
One row per category-week: keyword annotation totals plus the flattened structural, AI, thematic, and concern layers. The grain of every trend surface on the site.
| Field | Type | Description |
|---|---|---|
| id | integer | Database row id of the weekly aggregate. Stable within one database copy only — do not use as a cross-download key; (category, weekOf) is the natural key. |
| category | string | Key of one of the 14 monitored institutional categories (e.g. civilLiberties, executiveActions). The full list with display titles is in lib/data/categories.ts; category pages use the same keys. |
| weekOf | date | Monday (YYYY-MM-DD) anchoring the ISO week this row summarizes. Every weekly series in the system is Monday-anchored; a validation invariant (G2c) rejects off-grid dates. Weeks run Monday 00:00 UTC through Sunday. |
| totalSeverity | number | Sum of final keyword-severity scores across the week’s scored documents. Keyword severity is an ANNOTATION layer only — it never drives concern status (AI document review does). Useful as a rough activity-intensity signal. |
| documentCount | integer | Number of scored documents in the category-week: the count of document_scores rows, not of all stored documents. Only the counting population is scored — documents must have substantive content (≥100 chars), be retrieval-relevant, and (for court records) be inside the documented counting scope (see counting_scope). This rule is applied uniformly to ALL periods, past and present, so counts are directly comparable across collection-method changes. |
| avgSeverityPerDoc | number | totalSeverity / documentCount; 0 when the week has no scored documents. |
| captureProportion | number 0–1 | Share of the week’s scored documents containing at least one capture-tier keyword match (the most severe annotation tier). |
| driftProportion | number 0–1 | Share of scored documents with at least one drift-tier keyword match. |
| warningProportion | number 0–1 | Share of scored documents with at least one warning-tier keyword match (the mildest tier). |
| severityMix | number | Weighted blend of the three tier proportions summarizing how severe the week’s keyword annotations skew. Annotation layer only. |
| captureMatchCount | integer | Total capture-tier keyword matches summed across the week’s documents. |
| driftMatchCount | integer | Total drift-tier keyword matches summed across the week’s documents. |
| warningMatchCount | integer | Total warning-tier keyword matches summed across the week’s documents. |
| suppressedMatchCount | integer | Keyword matches removed by negation/context suppression rules (e.g. “no evidence of X”) and therefore excluded from the counts above. |
| topKeywords | string (comma-joined) | Most frequent matched keywords for the week, most common first. |
| structuralScore | number | Composite structural anomaly score: mean of the absolute per-dimension z-scores (structural_* columns). Descriptive context only. |
| structural_composite | number | Same composite as structuralScore, carried inside the structural detail; the two agree. |
| structural_volume | number (z-score) | Z-score of weekly scored-document count against the category’s cycle-matched baseline period (empirical mean/stddev; ±2 is notable, beyond ±3 unusual). Descriptive context only — does not drive concern_status. |
| structural_volume_raw | number | Raw value of weekly scored-document count for this week, before z-scoring. |
| structural_volume_baselineMean | number | Baseline mean of weekly scored-document count: the average weekly value across the cycle-matched baseline period (e.g. week 30 of a term compares to the same administration cycle-year of the Biden baseline). |
| structural_volume_baselineStdDev | number | Baseline standard deviation of weekly scored-document count; the denominator of structural_volume. Floored to avoid runaway z-scores when baseline variance is near zero. |
| structural_typeComposition | number (z-score) | Z-score of distribution of documents across source types (Jensen-Shannon divergence vs baseline mix) against the category’s cycle-matched baseline period (empirical mean/stddev; ±2 is notable, beyond ±3 unusual). Descriptive context only — does not drive concern_status. |
| structural_typeComposition_raw | number | Raw value of distribution of documents across source types (Jensen-Shannon divergence vs baseline mix) for this week, before z-scoring. |
| structural_typeComposition_baselineMean | number | Baseline mean of distribution of documents across source types (Jensen-Shannon divergence vs baseline mix): the average weekly value across the cycle-matched baseline period (e.g. week 30 of a term compares to the same administration cycle-year of the Biden baseline). |
| structural_typeComposition_baselineStdDev | number | Baseline standard deviation of distribution of documents across source types (Jensen-Shannon divergence vs baseline mix); the denominator of structural_typeComposition. Floored to avoid runaway z-scores when baseline variance is near zero. |
| structural_functionalDistribution | number (z-score) | Z-score of distribution across functional buckets like rulemaking/enforcement/personnel (JSD vs baseline) against the category’s cycle-matched baseline period (empirical mean/stddev; ±2 is notable, beyond ±3 unusual). Descriptive context only — does not drive concern_status. |
| structural_functionalDistribution_raw | number | Raw value of distribution across functional buckets like rulemaking/enforcement/personnel (JSD vs baseline) for this week, before z-scoring. |
| structural_functionalDistribution_baselineMean | number | Baseline mean of distribution across functional buckets like rulemaking/enforcement/personnel (JSD vs baseline): the average weekly value across the cycle-matched baseline period (e.g. week 30 of a term compares to the same administration cycle-year of the Biden baseline). |
| structural_functionalDistribution_baselineStdDev | number | Baseline standard deviation of distribution across functional buckets like rulemaking/enforcement/personnel (JSD vs baseline); the denominator of structural_functionalDistribution. Floored to avoid runaway z-scores when baseline variance is near zero. |
| structural_agencyActivity | number (z-score) | Z-score of distribution of documents across issuing agencies (JSD vs baseline) against the category’s cycle-matched baseline period (empirical mean/stddev; ±2 is notable, beyond ±3 unusual). Descriptive context only — does not drive concern_status. |
| structural_agencyActivity_raw | number | Raw value of distribution of documents across issuing agencies (JSD vs baseline) for this week, before z-scoring. |
| structural_agencyActivity_baselineMean | number | Baseline mean of distribution of documents across issuing agencies (JSD vs baseline): the average weekly value across the cycle-matched baseline period (e.g. week 30 of a term compares to the same administration cycle-year of the Biden baseline). |
| structural_agencyActivity_baselineStdDev | number | Baseline standard deviation of distribution of documents across issuing agencies (JSD vs baseline); the denominator of structural_agencyActivity. Floored to avoid runaway z-scores when baseline variance is near zero. |
| structural_publicationTempo | number (z-score) | Z-score of day-of-week publication variance against the category’s cycle-matched baseline period (empirical mean/stddev; ±2 is notable, beyond ±3 unusual). Descriptive context only — does not drive concern_status. |
| structural_publicationTempo_raw | number | Raw value of day-of-week publication variance for this week, before z-scoring. |
| structural_publicationTempo_baselineMean | number | Baseline mean of day-of-week publication variance: the average weekly value across the cycle-matched baseline period (e.g. week 30 of a term compares to the same administration cycle-year of the Biden baseline). |
| structural_publicationTempo_baselineStdDev | number | Baseline standard deviation of day-of-week publication variance; the denominator of structural_publicationTempo. Floored to avoid runaway z-scores when baseline variance is near zero. |
| structural_sourceConvergence | number (z-score) | Z-score of share of the week’s activity where government and independent sources cover the same stories against the category’s cycle-matched baseline period (empirical mean/stddev; ±2 is notable, beyond ±3 unusual). Descriptive context only — does not drive concern_status. |
| structural_sourceConvergence_raw | number | Raw value of share of the week’s activity where government and independent sources cover the same stories for this week, before z-scoring. |
| structural_sourceConvergence_baselineMean | number | Baseline mean of share of the week’s activity where government and independent sources cover the same stories: the average weekly value across the cycle-matched baseline period (e.g. week 30 of a term compares to the same administration cycle-year of the Biden baseline). |
| structural_sourceConvergence_baselineStdDev | number | Baseline standard deviation of share of the week’s activity where government and independent sources cover the same stories; the denominator of structural_sourceConvergence. Floored to avoid runaway z-scores when baseline variance is near zero. |
| structural_anomalous | boolean | True when the composite exceeds the anomaly threshold — a flag for “this week’s document-flow shape departs from baseline”, not a concern finding. |
| structural_driftTrend | string | Direction of the composite over recent weeks: rising, falling, or stable. |
| structural_longHorizon_cumulativeDeviation | number | Sum of composite deviation over the long-horizon window — surfaces slow sustained shifts a single week never would. |
| structural_longHorizon_cumulativeWindow | integer (weeks) | Length of the long-horizon window the cumulative deviation is summed over. |
| structural_functionalShifts | string (bucket:direction pairs) | Functional buckets whose share moved materially vs baseline, e.g. “enforcement:up, rulemaking:down”. |
| aiScore | number (z-score) | Z-score of the week’s AI Pass-1 flag rate against the category baseline. Summary of the detection layer; the classification detail is in the ai_* columns. |
| ai_flagCount | integer | Documents flagged by AI Pass 1 (screening model) as potentially erosion-relevant this week. |
| ai_totalDocuments | integer | Documents the AI review pipeline evaluated this week — the L2 evidence population. Can differ from documentCount: review eligibility (content ≥100 chars, retrieval-relevant) is not restricted by counting_scope, so out-of-scope court opinions are still reviewed as evidence. |
| ai_flagRate | number 0–1 | ai_flagCount / ai_totalDocuments. |
| ai_concernRate | number 0–1 | Share of Pass-2-reviewed documents classified potentially or clearly concerning. The primary driver of concern_status. |
| ai_p2_routine | integer | Pass-2 classifications: routine government activity. |
| ai_p2_novelNotConcerning | integer | Pass-2 classifications: novel but not concerning. |
| ai_p2_potentiallyConcerning | integer | Pass-2 classifications: potentially concerning — contributes to ai_concernRate. |
| ai_p2_clearlyConcerning | integer | Pass-2 classifications: clearly concerning — contributes to ai_concernRate. |
| ai_auditFalseNegativeRate | number 0–1 | From the audit sample of Pass-1-UNflagged documents sent to Pass 2 anyway: the share Pass 2 would have flagged. Measures what screening misses. |
| thematicScore | number (z-score) | Same value as thematic_zScore, surfaced at the top level. |
| thematic_centroidDistance | number 0–2 | Cosine distance between this week’s document-embedding centroid and the rolling 8-week centroid. Higher = the week’s topics moved further from the recent norm. Computed over the counting population. |
| thematic_zScore | number (z-score) | centroidDistance z-scored against the rolling 8-week window’s own mean/stddev. Mean-reverting by construction — single-week spikes matter more than long runs. Descriptive context only. |
| thematic_novelDocRate | number 0–1 | Share of the week’s documents farther than the novelty threshold (calibrated to the 90th percentile of doc-to-centroid distances) from the rolling centroid. |
| thematic_varianceRatio | number | This week’s embedding variance over the rolling window’s variance: >1 = topics more scattered than usual, <1 = more concentrated. |
| thematic_crossAdminDistance | number 0–2 | Distance between this week’s centroid and the cycle-matched baseline administration’s centroid — how far topics sit from the prior administration’s norm. |
| thematic_rollingWindow_weeks | integer | Weeks actually available in the rolling window (up to 8; fewer near a term start). |
| thematic_rollingWindow_meanDistance | number | Mean weekly centroid distance across the rolling window — the baseline thematic_zScore compares against. |
| thematic_rollingWindow_stdDev | number | Standard deviation of the rolling window’s distances — thematic_zScore’s denominator. |
| thematic_crossAdminBaseline | string | Identifier of the baseline period used for crossAdminDistance (e.g. biden_2022). |
| thematic_bootstrap | boolean | True in early-term weeks where the rolling window is too short for reliable statistics — treat thematic values as low-confidence. |
| convergenceScore | integer | Count of layers reading elevated this week (the concern_*Elevated flags). Context only; concern_status is not a function of this count. |
| concern_status | string enum | Convergence status for the category-week: Stable, Elevated, or ConfirmedConcern. Driven solely by AI document review (two-pass, cross-provider); structural, silence, and thematic layers are descriptive context and cannot change this value. Verified comparable across all pipeline changes by zero-flip gates. |
| concern_pattern | string | Short machine-generated description of which layers are elevated and how they align. |
| concern_structuralElevated | boolean | Structural composite above its elevation threshold this week (context flag). |
| concern_aiElevated | boolean | AI review layer elevated — the flag that drives concern_status. |
| concern_silenceElevated | boolean | Silence detection found government sources conspicuously quiet while independent sources stayed active (context flag). |
| concern_thematicElevated | boolean | Thematic drift above its elevation threshold this week (context flag). |
| computedAt | timestamp | When the aggregate row was last computed. Repairs and re-derivations refresh it. |
Document Scores CSV
One row per scored document: keyword annotation detail. Scored documents are the counting population — see documentCount in the weekly CSV for the rules.
| Field | Type | Description |
|---|---|---|
| id | integer | Database row id of the score row (single-copy stable only; use url + category as the natural key). |
| documentId | integer|empty | Row id of the scored document in the documents table, when linked. |
| url | string | Source URL of the document — the stable join key to the documents table (with category). |
| category | string | Key of one of the 14 monitored institutional categories (e.g. civilLiberties, executiveActions). The full list with display titles is in lib/data/categories.ts; category pages use the same keys. |
| severityScore | number | Weighted sum of the document’s keyword matches (tier weights: capture > drift > warning), before the class multiplier. Annotation layer only — never drives concern status. |
| finalScore | number | severityScore × classMultiplier — the value weekly totalSeverity sums. |
| captureCount | integer | Count of capture-tier keyword matches in the document (after negation and suppression filtering). Tiers by increasing severity: warning, drift, capture; a capture match carries weight highest in severity_score. |
| driftCount | integer | Count of drift-tier keyword matches in the document (after negation and suppression filtering). Tiers by increasing severity: warning, drift, capture; a drift match carries weight middle in severity_score. |
| warningCount | integer | Count of warning-tier keyword matches in the document (after negation and suppression filtering). Tiers by increasing severity: warning, drift, capture; a warning match carries weight lowest in severity_score. |
| suppressedCount | integer | Matches removed by negation/suppression rules for this document. |
| documentClass | string | Document class assigned by the classifier (e.g. official_action, press_release, court_document) used to weight keyword severity — official actions weigh more than commentary. The multiplier applied is class_multiplier. |
| classMultiplier | number | Weight applied to severityScore for the document class (official actions above commentary). |
| isHighAuthority | boolean | Source is a high-authority issuer (courts, GAO, inspectors general). |
| matches_count | integer | Number of keyword matches kept after suppression (equals captureCount + driftCount + warningCount). |
| matches_keywords | string (comma-joined) | The matched keywords themselves. |
| suppressed_count | integer | Number of suppressed matches (equals suppressedCount). |
| suppressed_keywords | string (comma-joined) | Keywords whose matches were suppressed, with the applicable rule recorded in the full dump’s JSON. |
| scoredAt | timestamp | Timestamp when the keyword scorer produced this row. Re-scoring (repairs, rule changes) refreshes it; it is not the document’s publication time. |
| weekOf | date | Monday (YYYY-MM-DD) anchoring the ISO week this row summarizes. Every weekly series in the system is Monday-anchored; a validation invariant (G2c) rejects off-grid dates. Weeks run Monday 00:00 UTC through Sunday. |
tracked_cases (dump)
One row per tracked federal case: the case→category routing universe (191,800 cases) joined with CourtListener docket metadata (filing/termination dates, status) and an optional cached posture line. Seeded from CourtListener bulk docket data plus our historical docket-stub rows; refreshed weekly for open cases. Format change 2026-08: this table replaces the ~283k metadata-only docket-stub rows formerly in documents — the case universe and activity dates now live here.
| Field | Type | Description |
|---|---|---|
| id | serial | Row id (stable within a single dump only). |
| case_id | varchar | cl:<docketId> — the same identifier documents.case_id carries for CourtListener rows; unique here (one row per case). |
| docket_id | bigint | CourtListener docket primary key (numeric form of case_id). |
| categories | jsonb | Monitored categories this case is routed to (string array). The authoritative case→category mapping — previously recoverable only from metadata-only docket-stub document rows. |
| case_name | text | Case caption (bulk-authoritative when available, else as first observed). |
| court_id | varchar|null | CourtListener court identifier (e.g. dcd, ca9, scotus). |
| court_name | varchar|null | Human-readable court name for display (from bulk court data). |
| docket_number | varchar|null | Court docket number as assigned by the filing court. |
| nature_of_suit | varchar|null | PACER nature-of-suit string (e.g. "440 Civil rights other"). |
| cause | varchar|null | PACER cause-of-action string (statute and claim shorthand). |
| date_filed | date|null | Case filing date per CourtListener bulk data or API refresh. |
| date_terminated | date|null | Docket termination date; null while the case is open. |
| date_last_filing | date|null | Most recent filing date CourtListener has recorded. |
| status | varchar | 'open' or 'terminated' — derived from date_terminated. |
| posture | jsonb|null | Cached one-line case posture from the live docket-timeline fetch: { line, eventType, date, asOf }. asOf is the CourtListener fetch time — the data age. |
| case_summary | text|null | Plain-language description of what the case is about, copied from the latest Pass-2 AI review reasoning of an opinion in this case (ai_document_assessments.reasoning) — AI-generated analysis, not a finding of fact. Present only for cases with an AI-assessed opinion (~4% of cases, concentrated on erosion-relevant litigation); synced weekly. |
| cluster_disposition | text|null | Latest opinion cluster's disposition text (bulk-sourced). |
| cluster_precedential | varchar|null | Latest opinion cluster precedential status (published/unpublished). |
| cluster_citation_count | integer|null | Citation count of the latest opinion cluster on this docket. |
| provenance | jsonb|null | How the case entered the universe: union of CourtListener query provenance markers (e.g. 'scotus-all', 'circuits-exec', 'dcd-exec') plus 'stub-seed' (historical seed) or 'ingest' (weekly discovery). |
| first_seen_at | timestamptz|null | Earliest ingestion touch across the historical seed rows. |
| last_seen_at | timestamptz|null | Most recent ingestion touch (weekly discovery or seed). |
| refreshed_at | timestamptz|null | Last CourtListener API refresh; null = bulk/seed data only. |
| created_at | timestamptz | When this tracked-case row was first created. |
documents (dump)
Every ingested source document with full text and lineage flags. The flags matter: content_type, retrieval_relevant, and counting_scope define which rows the statistics describe. A document appears once per category that fetched it (url + category is the natural key); routing follows the signal definitions in lib/data/categories.ts. Two sources use derived routing rules: DHS OIG reports also appear under Immigration Enforcement when the component tag assigned by DHS OIG (stored in metadata.dhsComponents) is ICE, CBP, or USCIS, OR when the report title matches ICE/CBP/USCIS (case-sensitive) or border/immigra-/detention/detainee/deportation/asylum/287(g)/migrant/unaccompanied/correctional facility or center/processing center/ports of entry/alien/expedited removal (case-insensitive). Congressional documents (CREC floor speeches, CHRG hearing transcripts) are routed to every category whose topic terms (lib/data/topic-routing-terms.ts) match — hearings classify on the title plus the first 6,000 characters of the transcript, with hearing-specific term calibrations documented in that file; hearings matching no category are recorded in chrg_seen_ledger rather than ingested. Opinion rows carry a case_id (cl:<docketId>, the CourtListener docket key) joining them to tracked_cases, which holds case-level metadata. Format change 2026-08: ~283k metadata-only docket-entry stub rows were removed from this table; the case universe they indexed now lives in tracked_cases.
| Field | Type | Description |
|---|---|---|
| id | serial | Row id (single-copy stable only). The natural key is (url, category) — one document can appear under several categories. |
| source_type | varchar | Kind of document as ingested: e.g. judicial_opinion, press_release (DOJ API or DHS/ICE/CBP newsrooms — distinguish by source_origin), bill, floor_speech, or a Federal Register type like Rule / Notice. Format change 2026-08: court_opinion (docket-entry stub) rows were retired — the case universe they carried moved to tracked_cases. |
| category | varchar | Key of one of the 14 monitored institutional categories (e.g. civilLiberties, executiveActions). The full list with display titles is in lib/data/categories.ts; category pages use the same keys. |
| title | text | Document title as published by the source. |
| content | text|null | Full text as fetched (PDF-extracted where the source publishes PDFs). Stored raw — boilerplate stripping happens at assessment time only. An upsert guard never replaces substantive content (≥100 chars) with a near-empty refetch. |
| url | text | Source URL; with category, the natural key used by every derived table. |
| published_at | timestamptz|null | Publication time claimed by the source. Drives week assignment (Monday of its ISO week). |
| fetched_at | timestamptz | When this row was last written by ingestion (refreshed on re-fetch upserts). |
| metadata | jsonb|null | Source-specific fields: agency (issuing agency or human-readable court name), action, suitNature (court NOS code), caseId, clQueries (CourtListener query provenance), and others varying by source. |
| source_origin | varchar|null | Ingestion pipeline that produced the row: federal_register, courtlistener, doj, govinfo, govinfo_cpd, crec, chrg (congressional hearing transcripts, routed by title + opening-statement topic classification), oig, fec, legiscan, dhs_press (DHS/ICE/CBP newsroom scrape; CBP restricted to national media releases — the port-level local-media-release URL class is excluded at fetch, owner decision 2026-08-07; cross-host HQ mirrors deduped by normalized title + day with ICE > CBP > DHS precedence; the ICE HSI criminal-investigation subset also stores to lawEnforcement). Legacy origins (whitehouse, gdelt) remain stored but are excluded from all analysis. |
| content_type | varchar | full_text, or metadata_only for rows that intentionally carry no body (GDELT rhetoric records and unobtainable-body documents) — metadata_only rows are excluded from counts, statistics, embeddings, and search. Format change 2026-08: the largest metadata_only population, docket-entry stubs, was retired to tracked_cases. |
| parent_id | integer|null | Fragment lineage (2026-08): set on rows split out of a multi-topic Congressional Record granule; points at the parent documents.id, which remains the single source for the origin URL and metadata. Fragments are searchable (embedded) but sit outside the counting population (counting_scope=false) and outside AI assessment — see the methodology page's Congressional Record granularity note. NULL for all non-fragment documents. |
| search_rank_vector | tsvector|null | Compact full-text ranking vector (2026-08, hybrid retrieval): title (weight A) plus the first 20k characters of content (weight B), maintained by a database trigger. Keyword MATCHING uses the full generated search_vector; this column exists so ts_rank ordering never detoasts multi-megabyte vectors. Derived — omitted from the public dump and rebuilt by the trigger on restore. |
| case_id | varchar|null | Court-case identifier used to group filings and dedupe case-level counting. For CourtListener rows the format is cl:<docketId>, where <docketId> is the CourtListener docket primary key — joinable to https://www.courtlistener.com/docket/<docketId>/, the v4 API (/api/rest/v4/docket-entries/?docket=<docketId>), and tracked_cases.case_id (the case-level record: court, dates, status, posture). Every CourtListener opinion carries case_id. |
| speaker | varchar|null | Attributed speaker for floor speeches (CREC). |
| embedding | vector|null | Text embedding (pgvector) used by thematic drift and semantic search. Computed only for the counting population. |
| embedded_at | timestamptz|null | When the embedding was computed; also set as an attempted marker when embedding failed so the pipeline does not retry forever. NULL = not yet embedded (or re-queued after a content repair). |
| retrieval_relevant | boolean|null | NULL/true = relevant. false = annotated off-topic for its category by the retrieval-relevance filter; kept for auditability but excluded from assessment, statistics, search, and exports of derived values. |
| counting_scope | boolean|null | NULL/true = inside the counting population. false = a court-category judicial opinion outside the documented counting rule (classifier v1: every SCOTUS opinion; circuit/D.D.C. opinions containing executive-power phrases). Applied uniformly to ALL eras so document counts are method-consistent across the February 2026 collection change. Out-of-scope opinions stay stored and remain AI-review evidence — this flag governs counting only. |
| search_vector | tsvector (database-managed) | Generated full-text search index over title/content. Maintained by the database, not application code; not present in the Drizzle schema. |
document_scores (dump)
Per-document keyword annotation rows; the JSON match detail the CSV flattens.
| Field | Type | Description |
|---|---|---|
| id | integer | Database row id of the score row (single-copy stable only; use url + category as the natural key). |
| document_id | integer|empty | Row id of the scored document in the documents table, when linked. |
| url | string | Source URL of the document — the stable join key to the documents table (with category). |
| category | string | Key of one of the 14 monitored institutional categories (e.g. civilLiberties, executiveActions). The full list with display titles is in lib/data/categories.ts; category pages use the same keys. |
| severity_score | number | Weighted sum of the document’s keyword matches (tier weights: capture > drift > warning), before the class multiplier. Annotation layer only — never drives concern status. |
| final_score | number | severityScore × classMultiplier — the value weekly totalSeverity sums. |
| capture_count | integer | Count of capture-tier keyword matches in the document (after negation and suppression filtering). Tiers by increasing severity: warning, drift, capture; a capture match carries weight highest in severity_score. |
| drift_count | integer | Count of drift-tier keyword matches in the document (after negation and suppression filtering). Tiers by increasing severity: warning, drift, capture; a drift match carries weight middle in severity_score. |
| warning_count | integer | Count of warning-tier keyword matches in the document (after negation and suppression filtering). Tiers by increasing severity: warning, drift, capture; a warning match carries weight lowest in severity_score. |
| suppressed_count | integer | Matches removed by negation/suppression rules for this document. |
| document_class | string | Document class assigned by the classifier (e.g. official_action, press_release, court_document) used to weight keyword severity — official actions weigh more than commentary. The multiplier applied is class_multiplier. |
| class_multiplier | number | Weight applied to severityScore for the document class (official actions above commentary). |
| is_high_authority | boolean | Source is a high-authority issuer (courts, GAO, inspectors general). |
| scored_at | timestamp | Timestamp when the keyword scorer produced this row. Re-scoring (repairs, rule changes) refreshes it; it is not the document’s publication time. |
| week_of | date | Monday (YYYY-MM-DD) anchoring the ISO week this row summarizes. Every weekly series in the system is Monday-anchored; a validation invariant (G2c) rejects off-grid dates. Weeks run Monday 00:00 UTC through Sunday. |
| matches | jsonb | Kept keyword matches with tier, weight, and a text snippet of surrounding context for each (the CSV flattens this to counts + keywords). |
| suppressed | jsonb | Suppressed matches with the negation/suppression rule and reason each one triggered. |
weekly_aggregates (dump)
The weekly CSV’s source table, with layer detail as JSON plus lineage stamps.
| Field | Type | Description |
|---|---|---|
| category | string | Key of one of the 14 monitored institutional categories (e.g. civilLiberties, executiveActions). The full list with display titles is in lib/data/categories.ts; category pages use the same keys. |
| id | serial | Row id. Natural key: (category, week_of). |
| week_of | date | Monday (YYYY-MM-DD) anchoring the ISO week this row summarizes. Every weekly series in the system is Monday-anchored; a validation invariant (G2c) rejects off-grid dates. Weeks run Monday 00:00 UTC through Sunday. |
| total_severity | real | Sum of final keyword-severity scores across the week’s scored documents. Keyword severity is an ANNOTATION layer only — it never drives concern status (AI document review does). Useful as a rough activity-intensity signal. |
| document_count | integer | Number of scored documents in the category-week: the count of document_scores rows, not of all stored documents. Only the counting population is scored — documents must have substantive content (≥100 chars), be retrieval-relevant, and (for court records) be inside the documented counting scope (see counting_scope). This rule is applied uniformly to ALL periods, past and present, so counts are directly comparable across collection-method changes. |
| avg_severity_per_doc | real | total_severity / document_count (0 for empty weeks). |
| capture_proportion | real | Share of scored documents with ≥1 capture-tier match. |
| drift_proportion | real | Share of scored documents with ≥1 drift-tier match. |
| warning_proportion | real | Share of scored documents with ≥1 warning-tier match. |
| severity_mix | real | Weighted blend of tier proportions (annotation layer). |
| capture_match_count | integer | Total capture-tier matches across the week. |
| drift_match_count | integer | Total drift-tier matches across the week. |
| warning_match_count | integer | Total warning-tier matches across the week. |
| suppressed_match_count | integer | Matches removed by suppression rules across the week. |
| top_keywords | jsonb | Most frequent matched keywords with counts (CSV joins these to a string). |
| structural_score | real|null | Composite structural anomaly score (see the weekly CSV structural_* columns for the flattened detail). |
| structural_detail | jsonb|null | Full structural result: per-dimension z-scores with raw/baseline stats, anomaly flag, drift trend, long-horizon accumulation, functional shifts. The CSV structural_* columns are this object flattened. |
| thematic_score | real|null | Thematic drift z-score (see thematic_* CSV columns). |
| thematic_detail | jsonb|null | Full thematic result: centroid distance, novelty rate, variance ratio, rolling-window stats, cross-admin comparison, bootstrap flag. |
| convergence_score | integer|null | Count of elevated layers (context only). |
| convergence_detail | jsonb|null | Concern synthesis: status (drives everything status-colored in the UI), pattern description, per-layer elevation flags. |
| ai_score | real|null | AI flag-rate z-score (see ai_* CSV columns). |
| ai_detail | jsonb|null | AI review summary: flag counts and rates, Pass-2 classification distribution, audit false-negative rate. |
| computed_at | timestamptz | When the aggregate was last recomputed. |
| enriched_at | timestamptz|null | Lineage stamp: when the enrichment layers (structural/AI summary/thematic/convergence) last ran for this row. NULL on legacy rows enriched before the stamp existed (mid-2026); validation treats those as a warning, not an error. |
ai_document_assessments (dump)
Per-document AI review rows — the sole active detection layer. Both passes, full reasoning, citations, audit samples, and per-call cost fields.
| Field | Type | Description |
|---|---|---|
| id | serial | Row id. Natural key: (url, category, pass). |
| document_id | integer|null | Row id of the assessed document, when linked. |
| url | text | Assessed document URL (join to documents with category). |
| category | varchar | Key of one of the 14 monitored institutional categories (e.g. civilLiberties, executiveActions). The full list with display titles is in lib/data/categories.ts; category pages use the same keys. |
| pass | integer | 1 = screening pass (flags potentially erosion-relevant documents), 2 = detailed classification of flagged documents. Different model providers per pass for epistemic independence. |
| relevant | boolean|null | Pass 1: whether the document was flagged as potentially erosion-relevant. |
| confidence | real|null | Model-reported confidence for its judgment (0–1). |
| erosion_type | varchar|null | Pass 2 erosion taxonomy: e.g. formal_override, operational_hollowing, norm_erosion — or null when not concerning. |
| erosion_actor | varchar|null | Attributed acting institution (e.g. fed_exec, state_local, judiciary) from the actor-attribution pass — distinguishes federal-executive action from other actors. |
| signals | jsonb|null | Structured signals the model cited (named patterns with short evidence). |
| assessment | varchar|null | Pass 2 classification: routine, novel_not_concerning, potentially_concerning, clearly_concerning. Weekly concern rates aggregate this field. |
| reasoning | text|null | Model’s stated reasoning for the classification. |
| comparative_context | text|null | Pass 2: how the action compares to prior-administration norms. |
| cited_passages | jsonb|null | Verbatim passages from the document the model cited as evidence. |
| counter_arguments | text|null | Model-generated strongest case AGAINST its own classification (skeptic discipline). |
| is_audit_sample | boolean | True when a Pass-1-unflagged document was deliberately sent to Pass 2 as an audit of screening recall — these rows power the audit false-negative rate. |
| model | varchar | Model identifier that produced the row (e.g. gpt-4o-mini for Pass 1, a Claude model for Pass 2). |
| provider | varchar | Model provider (openai / anthropic) — cross-provider by design. |
| prompt_version | varchar | Version tag of the prompt in force. Compare rates across prompt versions with care — calibration changes shift flag rates (documented in the methodology). |
| tokens_input | integer|null | Input tokens billed for the call (cost transparency). |
| tokens_output | integer|null | Output tokens billed for the call. |
| latency_ms | integer|null | Wall-clock latency of the model call, in milliseconds. |
| week_of | date | Monday (YYYY-MM-DD) anchoring the ISO week this row summarizes. Every weekly series in the system is Monday-anchored; a validation invariant (G2c) rejects off-grid dates. Weeks run Monday 00:00 UTC through Sunday. |
| assessed_at | timestamptz | When the assessment ran. Downstream freshness invariants compare enrichment and narrative timestamps against the newest assessed_at in each week. |
baselines (dump)
Per-category statistics for each prior-administration cycle-year baseline period.
| Field | Type | Description |
|---|---|---|
| id | serial | Row id. Natural key: (baseline_id, category). |
| baseline_id | varchar | Baseline period identifier, e.g. biden_2022 or trump_2018 — one prior-administration cycle year. Current-term weeks compare against the cycle-matched baseline. |
| category | varchar | Key of one of the 14 monitored institutional categories (e.g. civilLiberties, executiveActions). The full list with display titles is in lib/data/categories.ts; category pages use the same keys. |
| avg_weekly_severity | real | Mean weekly total_severity across the baseline period (annotation layer reference). |
| stddev_weekly_severity | real | Standard deviation of weekly severity across the baseline period. |
| avg_weekly_doc_count | real | Mean weekly scored-document count in the baseline period, under the same uniform counting rules as every other period. |
| avg_severity_mix | real | Mean weekly severity_mix across the baseline period. |
| drift_noise_floor | real|null | Baseline week-to-week centroid movement (mean + stddev): the amount of thematic drift that is normal for this category. Null when the period lacks embeddings. |
| embedding_centroid | vector|null | Mean embedding of the baseline period’s counting-population documents — the reference point for cross-administration thematic distance. |
| cycle_year | integer|null | Year within the presidential term (1–4) this baseline covers; used for cycle-matched comparison. |
| administration | varchar|null | Administration the baseline period belongs to (biden / trump). |
| calendar_year | integer|null | Calendar year the period mostly covers, for convenience. |
| computed_at | timestamptz | When baseline statistics were last recomputed (re-derivations refresh all baselines under current rules). |
narratives (dump)
AI-generated weekly and term narrative texts with generation provenance.
| Field | Type | Description |
|---|---|---|
| id | serial | Row id. Natural key: (category, week_of, version). |
| category | varchar | Key of one of the 14 monitored institutional categories (e.g. civilLiberties, executiveActions). The full list with display titles is in lib/data/categories.ts; category pages use the same keys. |
| week_of | date | Monday (YYYY-MM-DD) anchoring the ISO week this row summarizes. Every weekly series in the system is Monday-anchored; a validation invariant (G2c) rejects off-grid dates. Weeks run Monday 00:00 UTC through Sunday. |
| version | varchar | Narrative audience/format version (expert and public variants). |
| content | text | The generated narrative text (markdown). |
| model | varchar | Model that generated the narrative (multi-pass: draft and final are Claude, critique is GPT). |
| generated_at | timestamptz | When this text was generated. Never rewritten — it is provenance. A narrative can be older than later assessment data for its week; see staleness_accepted_at. |
| staleness_accepted_at | timestamptz|null | Owner’s explicit accept-as-is decision after reviewing post-generation assessment changes for the week (repairs/backfills routinely extend past weeks’ assessment sets without changing statuses). Assessment data newer than this stamp re-flags the narrative in validation. Null = never needed acceptance, or staleness not yet reviewed. |