Skip to content
JordanCoinPublic

About

Local-first FOIA automation with AI-powered document analysis. Your data never leaves your machine. πŸ”’πŸ‡ΊπŸ‡Έ

Topics

Resources

Stars

14 stars

Watchers

1 watching

Forks

Repository files navigation

OpenFOIA

Local-first investigation toolkit for journalists, researchers, and citizens.

Anyone, anywhere, should be able to investigate power safely. Most of the world has no legal right to demand records from its government, and where the right exists, asking can be dangerous. OpenFOIA is built for the reporter in Istanbul, the researcher in Nairobi, the citizen in SΓ£o Paulo: it runs on your machine, works offline, and can be wiped. Your data never leaves your machine unless you choose to send it.

One real government document, 45 entities, 166 relationships, extracted locally in seconds. No cloud. No API keys.

Entity relationship graph: 45 entities, 166 relationships extracted from a Palantir police surveillance contract pulled from DocumentCloud

Built-in document reader with entity highlighting. Click any entity to jump to it in the source text, with a link to the original document


Install

With Claude Code (recommended)

If you use Claude Code, install the OpenFOIA plugin. You get the CLI plus a built-in copilot that knows every command:

/plugin marketplace add JordanCoin/openfoia
/plugin install openfoia@openfoia
/foia-install

The plugin ships a skill that teaches Claude how to run FOIA investigations, plus slash commands for the core loop: /foia-search, /foia-investigate, /foia-graph, /foia-install. The last one bootstraps the CLI on your machine.

With any MCP client (Claude Desktop, Claude Code, others)

openfoia mcp serve runs OpenFOIA as a local MCP server over stdin/stdout (no network port). After openfoia install-extras mcp:

{ "mcpServers": { "openfoia": { "command": "openfoia", "args": ["mcp", "serve"] } } }

or claude mcp add openfoia -- openfoia mcp serve. Agents get openfoia_search (the openfoia search code path: off unless you add --allow-network, and every search needs a confirmed second call after the warning) and read-only local tools over your cases, stored documents, entities and requests. No tool deletes, purges, sends requests or messages, or changes config. Whatever a tool returns goes to the client's model provider. Setup and details: GUIDE.md, AI agents over MCP.

Shell install (no Claude Code needed)

curl -fsSL https://raw.githubusercontent.com/JordanCoin/openfoia/main/install.sh | bash

Works on Linux, macOS, and Windows (WSL).

New to OpenFOIA? Run openfoia guide after installing.

What It Does

Find the records. Read them page by page. Record findings with citations. Draft the request to the body that actually holds the record, under the law that body answers to, and track its clock. All locally, all offline, all purgeable.

openfoia init                                          # 53 federal agencies pre-loaded
openfoia case new "Gifted program, Detroit 1990s"      # one question, one case
openfoia jurisdiction show us-mi                       # the law a Michigan school district answers under
openfoia agency add --name "Detroit Public Schools Community District" -j us-mi -l school_district --email foia.request@detroitk12.org
openfoia records search "Palantir" --source documentcloud --json --rank police,contract   # search 10M+ public documents, ranked
openfoia records fetch --from hits.json                # pull every hit's text, page by page, into the case
openfoia records fetch CIA-RDP96-00789R002200230001-2 --source cia   # CIA reading room PDF, OCR/ingest locally
openfoia docs grep '"school board" OR WISC' -C 120     # every passage, with page numbers, offline
openfoia case note "No document names minors" --doc 3f2a --page 51 --quote "..." --tier says
openfoia analyze graph --name investigation --no-text --view   # graph scoped to the case
openfoia case export -f md -o report.md                # findings with citations, sources, query log
openfoia request mark REQ-2026-001 denied              # starts the appeal clock the statute sets

Features

PDF text extraction Compiled 2.6MB binary, zero dependencies. Resolves broken fonts and missing encodings without OCR. Reports each page as text layer, garbled or image only, and only the last two get OCR
Entity extraction Four tiers, LLM, GLiNER, spaCy, regex, each falling back to the next. Regex always works, with no model at all
Relationship graphs Interactive graph with built-in document reader. Click entities to read source text with highlights
Cross-reference MuckRock, DocumentCloud, OpenCorporates, SEC EDGAR, OpenSanctions, ICIJ Offshore Leaks
Claude Code skill The plugin teaches Claude the same loop with the same commands, so an agent and a human produce the same cited result
Cases and citations One case per question: searches, documents, findings (says / infer / speculation) with document and page, exported as a cited report
Any US jurisdiction Federal FOIA plus all 50 states and DC: statute, clock, fees, residency and appeal path per law, sourced from the RCFP Open Government Guide, with a verified flag and sources on every file. Add a school district, city, county or university as a public body under its state law
Request clocks Response, extension, tolling, appeal and still-interested deadlines computed from the law that applies, never from a federal default. deadlines check tells you what is overdue
File requests Email, fax (Twilio), physical mail (Lob) to 53 federal agencies or any body you add
Data interchange Import/export FollowTheMoney format (Aleph, OpenAleph, OpenSanctions)
Encrypted storage SQLCipher AES-256. Decoy profile mode
Forensic purge 3-pass overwrite, shell history scrub, free space fill
Portable mode openfoia portable keeps data, config and library caches on the USB; nothing is created in the host's home directory
Metadata stripping Strips image EXIF, PDF author and XMP, and DOCX author, company and revision data from the stored copy on docs ingest and web-UI upload; with mat2 installed (preinstalled on Tails) also OpenDocument, XLSX, PPTX, EPUB, audio and video. Every file is reported as stripped or not; a format nothing here can clean is flagged, never shown as done. Email attachments and archived web pages are stored as received; see THREAT_MODEL.md

PDF Extraction Engine

A compiled 2.6MB binary extracts text directly from PDF structures, so born-digital documents need no OCR. A multi-level resolution cascade handles broken fonts, missing encodings, and government form PDFs that other tools cannot read.

Measured with engine 0.8.0 on its test corpus of 176 real PDFs (15,808 pages): 13,456 pages (85 percent) extract as a clean text layer. 1,619 are flagged garbled and 733 image only; only those pages go to OCR (Tesseract), and the rest of the document keeps its exact text. Extraction is deterministic: the same file gives the same text on every run. Zero runtime dependencies.

An optional open-source font pack (12.6MB download, 24MB on disk) feeds the engine's fingerprint stage, which only engages for embedded fonts with no usable character map. On the same corpus with engine 0.8.0 it changes no page verdicts, so the installer skips it unless you run install.sh --with-fonts. It lands in ~/.local/share/glyph-api/fonts, or next to the binary in portable mode, and openfoia init reports whether it is present.

OpenFOIA calls the extractor with the large-document runtime profile by default (--profile large). To control concurrency per process, set:

  • OPENFOIA_PDF_EXTRACT_MAX_WORKERS (for example 4 or 8)
  • Optional profile override: OPENFOIA_PDF_EXTRACT_PROFILE=standard|large

Reading and searching stored documents

Everything below runs against the local database, with no network access.

openfoia docs list                                  # stored documents, pages, source
openfoia docs show 3f2a --page 12                   # one page of text (omit --page for all)
openfoia docs grep Hocking                          # every passage that mentions Hocking
openfoia docs grep '"school board" OR WISC' -C 80   # FTS5 phrases, OR, NEAR(a b, 10)
openfoia docs grep audit --doc 3f2a --json          # one document, machine-readable

Results cite real page numbers for DocumentCloud documents and ingested PDFs. Documents stored before page support, or whose page text could not be fetched, are kept as a single page (docs show flags this when the source page count is known). If your SQLite build lacks FTS5 (some SQLCipher builds), docs grep does a plain substring search instead and prints a notice.

Optional and experimental: docs grep --semantic "payments to the mayor's brother" ranks pages by meaning, so a page that says "funds transferred to a sibling of the city executive" can still turn up. It needs openfoia install-extras semantic and a one-time model download (openfoia docs embed-model, ~31 MB, asks first); after that it is offline. It finds leads, not proof of absence: on our test set it surfaced 3 of 10 reworded passages in the top 10. See the guide and the measurements.

Entity extraction picks the best backend it finds. On a local model the LLM validation step can take minutes per document. To skip it, force a local backend with openfoia analyze extract <id> --tier gliner (or spacy, regex), or set OPENFOIA_AI_PROVIDER=none, which wins over any API key or Ollama URL in the config file or environment.

Documents not in English: the GLiNER tier's default model is English-only. openfoia install-extras ner-multilingual and --ner-model multilingual (or OPENFOIA_NER_MODEL=multilingual) switch it to a multilingual model that finds far more names in Arabic, Hindi, Bengali, Swahili and other languages, at the cost of about 1 GB more RAM and a one-time 1.25 GB download. The default does not change; analyze extract suggests the switch when a document looks non-English. See the guide.

Each extraction is stored as a run (backend, model, time, entity count). --force adds a run instead of overwriting the last one; graphs, crossref and exports use the newest run per document. openfoia analyze runs <id> lists them, analyze extract <id> --prefer-run <run> picks another, and analyze runs <id> --prune deletes the rest.

Which law applies

Federal FOIA reaches only federal executive agencies. A school district, city, county, police department or public university answers under its state's public-records act, with that state's clock, fees and appeal path. OpenFOIA carries one record per law, built from the RCFP Open Government Guide, each with its sources and a verified flag. Unknown values stay unknown; nothing is invented.

openfoia jurisdiction list                        # 51 verified, 1 unverified
openfoia jurisdiction show us-tx                  # statute, clock, fees, residency, appeal path, sources
openfoia agency add --name "Austin ISD" -j us-tx -l school_district --custodian-title "Public Information Officer"
openfoia request status REQ-2026-001              # every clock with days left and the rule behind it
openfoia request mark REQ-2026-001 extended       # pushes the due date by the statute's extension
openfoia deadlines check                          # overdue responses, appeals, still-interested replies

Which rung of the system holds your record, and how to ask at each, is in The Records Ladder.

Data Sources

openfoia records search supports these public-records sources. Each search sends its query to the selected third party without a confirmation prompt, so treat queries as sensitive investigation data and run one only when you accept that disclosure.

Source What Access
MuckRock Completed FOIA requests with downloadable response documents API token (OPENFOIA_MUCKROCK_TOKEN)
DocumentCloud Public documents with pre-extracted text Public API
CIA FOIA Reading Room Declassified CIA records (CREST, MORI); records fetch <CREST or MORI> --source cia downloads and ingests the PDF Public site; a bot filter can block scripted access, reported as "blocked"
National Archives Catalog Accessioned federal records: descriptions, OCR text where NARA has it, and the scans themselves. records fetch <NAID> --source nara --objects downloads every page image or PDF of a record and OCRs it locally; records search --source nara --ancestor <NAID> --all lists every file unit in a series Free key by email (OPENFOIA_NARA_API_KEY), 10,000 queries a month, counted locally
CourtListener Federal court dockets, filings and parties from the RECAP archive of PACER, and case law. records fetch <docket id> --source courtlistener stores the docket sheet (entries, parties, attorneys); rd:<id> a filing's text, or its PDF when CourtListener has no text; op:<id> an opinion Free personal token (OPENFOIA_COURTLISTENER_API_KEY), environment variable only. 5 requests a minute, 50 an hour, 125 a day, counted locally. CourtListener logs searches against your account unless query logging is off in your profile; not used over Tor
OCCRP Aleph Company registries, leaks, court records, gazettes and other datasets OCCRP collects from more than 230 online sources (its focus: Central and Eastern Europe, Sub-Saharan Africa, South America and offshore jurisdictions), as FollowTheMoney entities. records search -s aleph filters by --type (person, company, organization, vessel, document, ...) and -j country code; records fetch <entity id> -s aleph stores the entity and its neighbours (directors, owners, other linked entities) and imports them into the entity graph Personal API key (OPENFOIA_ALEPH_API_KEY), environment variable only. Searching without a key was refused (HTTP 401) when checked on 2026-11-19. Accounts are reviewed by hand by OCCRP and need an authenticator app; leaks need extended access. The key identifies you to OCCRP; not used over Tor. OCCRP's terms allow journalism use only and ask you to credit "OCCRP Aleph Pro". OpenFOIA caps itself at 30 requests a minute and 500 a day
UK Companies House UK companies, officers (directors, secretaries), persons with significant control (owners) and filing history. records search -s companieshouse searches companies, or officers with -t officer; records fetch <company number> -s companieshouse stores the company, its officers, owners and filings and imports them into the entity graph; officer:<id> stores one person's appointments Free API key from a Companies House account (OPENFOIA_COMPANIES_HOUSE_API_KEY), environment variable only. The key belongs to your account, so it identifies you to Companies House; not used over Tor. 600 requests per 5 minutes, counted locally; a company fetch is 4 requests
OpenCorporates Global company registrations, ownership, directors, and filings Free basic search
SEC EDGAR US public-company filings No API key
USAspending Federal contracts, grants, loans, and other awards No API key
ProPublica Nonprofit Explorer IRS Form 990 filings and US nonprofit data No API key
GovInfo Congressional, court, regulatory, and other US government documents Free API key; demo key available
OpenFEC Federal campaign-finance contributions and committees Free API key; demo key available
Regulations.gov Federal rulemaking documents and public comments Free API key; demo key available
Federal Register Federal rules, proposed rules, notices and presidential documents, filterable by agency, type, publication date and docket, with comment deadlines. records fetch <document number> --source federalregister downloads the PDF from GovInfo and ingests it No API key
OpenSanctions Sanctions lists, politically exposed persons (PEPs), crime and debarment lists, worldwide. Each result names the lists (datasets), topics (sanction, PEP, crime...), countries and a match score, shown as confidence in openfoia search --json. records fetch <id> --source opensanctions stores the entity (FollowTheMoney-shaped) in the active case Offline once you run openfoia records opensanctions-data --download (asks first; about 480 MB from data.opensanctions.org, size and SHA-1 checked against the published index): openfoia search, records search, records fetch and crossref then screen names on this machine and send nothing. Without the file: the /match API with a personal key (OPENFOIA_OPENSANCTIONS_API_KEY, environment only; free for journalists, nonprofits and researchers, otherwise one paid credit per name), sent in a POST body; never used over Tor. Data license CC BY-NC 4.0: free for non-commercial use
ICIJ Offshore Leaks Panama, Pandora, and Paradise Papers Cross-reference source; local CSV download

Search every source at once

openfoia search sends one query to every records search source (or the ones named with --source) and reports each one as ok, no results, error (with the reason) or skipped (with the reason). It names the sources and the route (Tor or direct) and asks before anything is sent; --yes skips the question.

openfoia search "Acme Corp"
openfoia search "Acme Corp" --source sec --source opencorporates --limit 5
openfoia search "Acme Corp" --egress tor --json --yes > hits.json

Every result carries source, title, url, snippet, retrieved_at (UTC) and egress (tor or direct). --egress tor stops with an error when Tor is not running instead of going direct; without --egress the configured route (network.tor) is used. Over Tor, MuckRock and NARA are skipped because they only work with your personal key, which would identify you. --since YYYY-MM-DD drops results dated earlier after they arrive; the sources still get the same query. The agent interface has the same search as the search_records tool, and MCP clients as openfoia_search.

Documentation

Guide What
Journalist Guide Full walkthrough. Start here if you're new
The Records Ladder Which rung of the US public-records system holds your record, from the school district to NARA, and how to ask
Tails OS Running from Tails for maximum privacy
USB Install Encrypted USB portable setup
Air-Gapped Offline deployment overview

Privacy

Everything runs on 127.0.0.1. Token-authenticated: openfoia serve opens the browser with a login link that works once and expires after 60 seconds (press Enter in its terminal for a fresh one), and the session token never appears in a URL. Encrypted at rest (optional). Forensically purgeable. Works offline. Open source.

For sensitive investigations: openfoia serve --tor

Sending requests in Tor mode: request send sends email, fax and mail through Tor, or stops; it never falls back to a direct connection. --direct sends one request without Tor when your mail provider blocks Tor. Tor hides your IP address from the provider, not your account or the name on the request (Guide).

Portable mode: openfoia portable keeps data, config and caches on the USB and creates nothing in the host's home directory (the OS can still log activity; see USB Install).

When you're done: openfoia purge --secure --yes

Uninstall

curl -fsSL https://raw.githubusercontent.com/JordanCoin/openfoia/main/uninstall.sh | bash

Legal Context for Contributors

Writing this code is legal. The US DOJ stated in August 2025: "Merely writing code, without ill intent, is not a crime." Open source encryption is exempt from US export controls. The Tor Project, Signal, SecureDrop, and Tails set the precedent. More context.

Credits

Built by people who believe in freedom of information. Inspired by MuckRock, DocumentCloud, and the Reporters Committee for Freedom of the Press.

License

AGPL-3.0. Keep it open.


"Democracy dies in darkness. FOIA is how we turn on the lights."

About

Local-first FOIA automation with AI-powered document analysis. Your data never leaves your machine. πŸ”’πŸ‡ΊπŸ‡Έ

Topics

Resources

Stars

14 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages