Autopia

Autopia empirically tests HTTP APIs that ship an OpenAPI spec, discovers how they actually behave, verifies every finding by reproducing it, and serves the result as a versioned, queryable knowledge base that AI agents consult over MCP, and that a Claude Code plugin injects into an agent's context at the moment it is about to call one of those APIs.

It is the MVP of the "Discrimen Compendium" described in docs/compendium-spec.txt (the blueprint) and docs/compendium-stack.txt (stack choices). Read the spec's glossary first; the code uses those exact terms. HANDOFF.txt holds the running state and decisions.

Live: the hosted server is at https://autopia-mcp.onrender.com/mcp (per-user keys, see Identity); the database is a managed Postgres on Neon; a GitHub Actions schedule refreshes every target nightly (rules only) and weekly (with the model pass). The catalogue currently holds 16 APIs, including GitHub REST, the npm registry, PyPI, the Anthropic models endpoints, the Open-Meteo family, weather.gov, Binance, Frankfurter, Open Library, PokeAPI and Nager.Date.

What an agent gets

A behavior sheet per API: Markdown generated from verified claims only, ordered by value. Where the live API contradicts its own spec comes first, then rules the spec never states, then behavior common to every endpoint, then per endpoint: what responses look like, error-body shapes, pagination, validation, auth. Every line ends with the trial ids that back it. Examples of what the sheets record, all reproduced on the live APIs:

• GitHub silently applies since=not-a-date on commits as a filter that matches nothing (200, empty list), while issues rejects the same value with a 422 that spells out the format.
• GitHub accepts a missing token on public reads but serves the request from the 60/hour tier instead of 5000/hour; per_page is clamped to 100 (proved by the Link header).
• npm search accepts size=2500 against a documented maximum of 250, and does not type-check its score-weight parameters.
• The Anthropic /v1/models cursors must name an existing model id; an unknown id is a 400 as a cursor but a 404 as a path.

The pipeline

discover -> exercise -> aggregate -> cross-ref -> verify -> write (+ diff)
 spec       TRIALS      rules +      spec +       evidence   CLAIMS
 fetch,     (no LLM)    shape rules  siblings +   check +    + DIFFS
 hash,                  + one LLM    self-        reproduce
 trim                   pass         consistency

• Trials are pinned requests generated by the battery (src/worker/battery, version 1.3.0), a deterministic catalogue of edge cases: omit required, wrong type, enum outside the documented set, null and empty values, numeric boundaries (zero, negative, max int, below the minimum, just above and far above the maximum, 2048-character strings), duplicate parameters, unknown fields, bad and missing credentials (scheme-preserving), page zero, a huge page size, a page beyond the last, four date formats for date-time parameters (date only, no timezone, epoch, garbage), wrong content type, trailing slash, HEAD. No randomness anywhere, so two runs produce identical trial inputs. Every stored response keeps the lengths of its arrays before truncation, so list semantics survive storage.
• Claims come from three sources: deterministic rules over the trials (free), shape rules over successful responses (content type, timestamp formats, null-valued fields, pagination fields, rate-limit and cache headers), and one batched LLM pass for open-ended discovery. The LLM output is validated with ajv against src/schemas/claim.schema.json; invalid objects are discarded. Rules classify what came back, not just the status: a 200 on a bad date value is "applied as an impossible filter" when the list is empty where the comparison request had rows, "apparently ignored" when the counts match, "interpreted" otherwise. A page-size cap is claimed only when more items provably exist (a Link rel="next", a total field, or the documented maximum reached). Error-shape claims count only probes that sent a bad value, not auth, HEAD or content-type probes.
• Verification is the product: every claim cites trial ids and the status each showed. The evidence check is mechanical; reproduction re-runs the exact inputs and compares a normalized response signature (status, body kind, JSON shape, error message), never raw bytes, so live data can change without breaking reproduction. Non-reproducing claims are discarded. Stored claims are always verified = true.
• Divergences are claims that contradict a documented fact (required flag, type, enum, range). The money view. Silence in the spec is never a divergence: a rule the API enforces that the spec never states (parameters that must be sent together, implicit defaults) is stored under its own kind, undocumented_constraint.
• Claim kinds: required, type, auth, pagination, error_behavior, response_shape, undocumented_constraint, undocumented, other.
• Sibling comparison. Tools that share a category are compared probe by probe: the same rule on the same parameter name with the opposite verdict marks the claim divergence_type = sibling with a note naming the other tool.
• Carry-forward. Rules-only runs re-admit the previous run's LLM claims as candidates, remapped to tonight's trials by endpoint and probe, and send them through the same evidence check and reproduction. A carried claim is dropped when a fresh LLM claim on the same endpoint cites the same probes (the model restates findings in new words), and carried claims that duplicate each other keep the more confident one.
• Diffs compare claim sets across tool versions by stable claim_key. A tool gets a new version whenever its spec hash changes; large specs are stored trimmed to the exercised paths, with the hash still over the full document.

Quick start

npm install
npm run autopia -- run targets/open-meteo-forecast.json     # full pipeline (LLM pass if a key is set)
npm run autopia -- docs open-meteo-forecast                 # the behavior sheet
npm run autopia -- claims open-meteo-forecast --divergences
npm run autopia -- search "is latitude required"
npm test

Configuration lives in a local .env file (gitignored), read on startup:

Target files reference credentials as ${ENV_VAR} in headers, so a committed target never contains a key.

PGlite caveat: the default database is an embedded Postgres (PGlite) under data/pglite and only one process can open it at a time. Stop the worker before starting the MCP server or dashboard. Set DATABASE_URL to a real Postgres to lift that limit (same schema, no code change). npx tsx scripts/whichdb.ts says which one a command would use.

Serving the compendium

npm run mcp          # MCP server over stdio (register in Claude Code / any MCP client)
npm run mcp:http     # the same server over Streamable HTTP at http://localhost:4200/mcp
npm run dashboard    # read-only dashboard at http://localhost:4100 (local only; not deployed)

Register the stdio server with a client as npx tsx <repo>/src/server/mcp.ts, or point an HTTP-capable client at /mcp on the HTTP server (GET /healthz for liveness). The server ships an instructions block telling an agent when to consult it: before its first request in a session to a covered host, and after an API call fails or surprises it.

The tool argument of every tool accepts a tool name, a host, or the URL you are about to call (api.github.com, https://registry.npmjs.org/-/v1/search). The compendium is a directory, not a dump: get_tool_docs with a URL or path returns only the endpoint that path matches plus a one-line index of the tool's other endpoints, so an agent descends to what it needs instead of receiving everything. Every sheet ends with a coverage section: which endpoints of this API are probed, which other APIs exist (by host), what is not probed anywhere yet, and the tools to fetch more, so a reader of one sheet knows what else it can ask for. Evidence is compact (trial id, probe, status, request line) unless verbose=true asks for response previews. Stored request inputs are redacted before storage, so no credential can be served as evidence; npx tsx scripts/scan-stored-secrets.ts audits the database for anything that slipped through.

Claude Code plugin: the sheet at the moment of use

An agent only reads a sheet if something tells it to. The plugin in plugin/ makes that deterministic: a PreToolUse hook on Bash and WebFetch matches the command or URL against the covered hosts and injects the behavior sheet for that endpoint into the model's context before the call, and a PostToolUse hook injects the verified claims that match a 4xx/5xx when a call to a covered host fails. Both fail open on any error. It also registers the hosted MCP server for digging further and ships a skill carrying the same trigger text.

claude plugin install autopia --marketplace MaydayTrey/autopia
claude plugin configure autopia        # optional: a per-user key raises your quota; reads work without one

The hook fires only on URL-shaped mentions (https://host/... or host/path) and injects the brief sheet for the endpoint in that URL: spec contradictions, undocumented rules, auth, pagination and error behavior for that endpoint, tool-wide auth behavior, and a one-line index of the other endpoints. Once per host and endpoint per session; well under 5KB.

Project-level instructions. A line in a repository's CLAUDE.md or AGENTS.md is followed more reliably than any server instructions block. For a project that integrates with a covered API, add:

Before the first call to api.github.com (or registry.npmjs.org, pypi.org, api.anthropic.com) in a
session, read its verified behavior sheet: Autopia MCP tool get_tool_docs {"tool":"<host>","brief":true}.
On an unexpected 4xx/5xx or an empty result from one of those hosts, call search_claims with the host
and the symptom before retrying.

For a session-only trial from a checkout: claude --plugin-dir ./plugin. Until the key is configured, the session shows the autopia MCP server as failed to connect (401) and the hook injects a setup note instead of a sheet; both clear after claude plugin configure autopia and a new session. Details, state locations and the privacy note are in plugin/README.md.

Static pages: where agents already look

A web search for an error string should land on a page that cites reproduced trials, with no plugin and no MCP connection. npx tsx scripts/build-site.ts --out site --base <url> renders the compendium as a static site: a catalogue, one page per API, one per endpoint, an index of every observed error message with what triggers it, llms.txt (the AI-crawler index), a sitemap and robots.txt. Pages are the behavior sheets with the lightest HTML wrapping; every page has a plain-text .md twin and names the MCP call that returns the same thing live.

.github/workflows/pages.yml builds it from the hosted database daily after the refresh (and on demand) and publishes to GitHub Pages. One-time setup: repository Settings -> Pages -> Source: GitHub Actions. The site is at https://<owner>.github.io/<repo>/.

Keeping it current (automation)

npm run run-all                        # every targets/*.json in sequence, one summary line per tool
npm run run-all -- --no-llm --prune-days 14

run-all first processes pending user requests, then re-fetches each spec, creates a new tool version when the spec hash changes, re-exercises, re-verifies, and computes the diff against the previous version. It exits non-zero if any target fails. Targets that are unreachable are abandoned early rather than timing out trial by trial.

Schedule. .github/workflows/run-all.yml runs rules-only nightly (free); run-all-llm.yml runs with the model weekly; both read DATABASE_URL, ANTHROPIC_API_KEY and the built-in GITHUB_TOKEN from repository secrets and can be triggered by hand from the Actions tab. The model pass is the only cost that scales with targets; carry-forward keeps its claims alive between weekly runs. Retention: --prune-days N (or autopia prune N) deletes runs older than N days except the latest finished run of each version and any run a diff cites; --dry-run reports without deleting. Elsewhere, use whatever scheduler you have (Task Scheduler, cron, a scheduled job on your host) with the same command.

User-requested targets

Anyone with a key can ask the compendium to test an API it does not have yet, from inside their agent, with the request_target MCP tool. The request is validated on the spot: the spec is fetched and parsed, the battery is planned, and the user gets back how many endpoints and trials it will take. Scope is enforced at submission, not by review:

• public http(s) hosts only, for the spec and for every server it names (no localhost, private ranges, IP literals, or .internal-style names: the worker must never become a proxy into wherever it runs)
• keyless: targets with headers are refused; credentials are the contributor tier's job
• politeness we control: concurrency 1, at least 300ms between calls, at most 300 trials
• a free tool name, and a quota of 3 requests per key per day

Pending requests are processed rules-only at the start of the next run-all pass, or by hand with npm run autopia -- requests process. Once done, the stored target joins every later pass like a file target. get_request_status reports pending / running / done / failed; npm run autopia -- requests list shows them all.

Keyed targets: two routes

Server tier, credential in Actions secrets. github-rest and anthropic-api are keyed targets the server runs itself: the target file names the header as ${GITHUB_TOKEN} or ${ANTHROPIC_API_KEY}, the workflow passes the secret, stored trial inputs are redacted by header name, and reproduction restores the real value from the target in memory. The secret is on the wire to the API and nowhere else. The battery's fake credential for bad-auth probes is deliberately not redacted, so reproduction can tell it apart from the real key.

Contributor tier, credential never leaves the contributor. For APIs whose key cannot live in this repository's secrets, the same open-source worker runs on the contributor's machine with their API key, their model key and their own database, then uploads the result:

npm run autopia -- contribute my-keyed-api --to https://autopia-mcp.onrender.com --key ak_... --yes

What gets uploaded, by construction: claims, and evidence as status code plus response signature (a type shape with no values in it). No request inputs, no headers, no response bodies, no error-message text. Claim statements are scrubbed twice, locally and on the server: anything that looks like an email, a token, a bearer credential, a long hex string or a long identifier number drops that claim. Without --yes the command prints every statement it would upload and stops. Contributor tools are a visible trust tier: their sheet opens with a banner naming the contributor key prefix and the date the claims last reproduced, and states that this server has not re-verified them since. A tool this server verifies itself can never be overwritten by a contribution.

Hosting: managed Postgres + the HTTP MCP server

The embedded database is for one laptop. A hosted compendium uses a managed Postgres so the worker and the server can run at the same time, and runs the HTTP MCP server somewhere always on. The code path is identical: DATABASE_URL switches drivers, migrations apply on connect, and tests/integration/postgres-path.test.ts exercises it over the real Postgres wire protocol.

1. Database (Neon or Supabase, both include pgvector). Create a project, copy the connection string, and move an existing compendium across:

npx tsx scripts/db-export.ts data/export.json

DATABASE_URL="postgres://..." npx tsx scripts/db-import.ts data/export.json

Import preserves ids, skips rows that already exist, and resets the sequences, so it is safe to re-run. Put the same DATABASE_URL in .env and every command on the laptop now talks to the hosted database.

2a. Server on Render (free tier, no card; the current deployment). render.yaml is a Blueprint. In Render: New -> Blueprint -> select the repo. It builds the Dockerfile, asks for DATABASE_URL and generates AUTOPIA_MCP_TOKEN (copy it from the service's Environment tab). Free services sleep after ~15 idle minutes and wake on the first request, so expect a slow first call; the plugin's hook tolerates that by failing open and retrying on the next call. Render auto-deploys from main.

2b. Server on Fly.io (paid account, always-on). fly.toml and the Dockerfile are in the repo:

fly launch --copy-config --no-deploy

fly secrets set DATABASE_URL="postgres://..." AUTOPIA_MCP_TOKEN="$(openssl rand -hex 24)"

fly deploy

The server listens on /mcp with GET /healthz for the platform check. The image also contains the worker, so run-all can run as a scheduled machine or by hand. Railway works the same way from the same Dockerfile.

Identity: anonymous reads, keys for writes

Reads are open, within a budget. The seven read tools answer without any credential, under a per-IP limit (AUTOPIA_ANON_PER_MINUTE, default 30; AUTOPIA_ANON_PER_DAY, default 300) and a global daily cap (AUTOPIA_ANON_DAILY_CAP, default 5,000). Anonymous callers never receive response previews and their page sizes are capped. Read responses (sheets, catalogue, searches, claims) are served from an in-memory cache for ten minutes (AUTOPIA_CACHE_TTL_MS), so repeated anonymous calls do not reach the database. The defaults are sized for the free tiers underneath: 5,000 brief sheets a day is roughly 75 MB of egress and a few hundred database round trips, well inside Render's and Neon's free allowances; raise the numbers when the hosting can take it. When the cap is hit, anonymous callers get 429 with a retry-after; keyed callers are unaffected. Public API behavior does not need per-user identity to read. AUTOPIA_ANON_READS=0 restores key-only access. request_target and /contribute always need a key.

Keys give a caller its own quota, a name in the usage log, and the write tools:

npm run autopia -- keys create alice --quota 2000

The key (ak_...) is printed once; only its hash is stored. Users send it as Authorization: Bearer ak_.... keys list shows prefixes, quotas, and call counts; keys revoke <name-or-prefix> disables one; a key over its daily quota gets 429. Every tool call is logged with the key, the tool, the target asked about, success, and latency, and npm run autopia -- usage --days 7 summarizes it per key and per most-asked-about tool.

AUTOPIA_MCP_TOKEN, if set, works as an admin token (no quota; compared in constant time; calls logged without a key). AUTOPIA_MCP_OPEN=1 disables the anonymous limits entirely (tests only). Keys are issued by the maintainer today; self-service issuance is on the roadmap.

CLI

AUTOPIA_LOG=debug prints every trial and every verification decision. Helper scripts under scripts/: run-progress.ts (watch a run), fail-stale-runs.ts (mark interrupted runs), inspect-repro.ts and auth-probe-check.ts (debug reproduction and auth probes), plan-dry-run.ts, db-smoke.ts, whichdb.ts, scan-stored-secrets.ts.

Adding a target

Create targets/<name>.json (the file name is the tool name):

{
  "name": "my-api",
  "category": "weather",
  "spec_url": "https://example.com/openapi.yml",
  "include_paths": ["/v1/path"],
  "baseline": { "/v1/path": { "required_param": "known good value" } },
  "optional_values": { "/v1/path": { "optional_param": "known good value", "since": "2026-01-01T00:00:00Z" } },
  "headers": { "x-api-key": "${MY_API_KEY}" },
  "rate": { "concurrency": 1, "min_interval_ms": 500 },
  "rate_overrides": { "/v1/search": { "min_interval_ms": 2500 } },
  "server": "https://api.example.com",
  "max_trials": 300
}

• include_paths limits a large spec to the endpoints worth probing (GitHub's spec has 816 paths; the target exercises 7). The stored spec is trimmed to those paths plus the components they reference, while the version hash covers the full document.
• rate_overrides slows down individual paths with their own quota (GitHub search: 30/min). The executor also treats a 403 with x-ratelimit-remaining: 0 as rate limiting and sleeps until the reset.
• server supplies a base URL when the spec declares none; override_server: true replaces the declared one (a regional mirror).
• A target may point at a spec file in the repo instead of a URL. specs/ holds hand-authored OpenAPI descriptions for APIs that publish none (npm registry, PyPI, the Anthropic models endpoints); divergences reported against those mean "the API disagrees with our description", and the file header says so.
• targets/parked/ holds targets that are temporarily out (see its README for why).

Scope rule (v1): read-only GET endpoints of OpenAPI 3.x APIs, public or behind a single credential header. Write endpoints and JSON request bodies are not probed yet. If a target lacks a formal spec or has side effects, it is out of scope; discover stops with an error.

Testing

npm test runs everything (130 tests); npm run test:unit and npm run test:integration split them.

• Unit, one file per stage under tests/unit: spec parsing, battery, signature, HTTP executor (fake fetch), rules, shape rules, cross-ref, carry-forward, verify, diff, docs rendering, redaction, keys, contribution scrubbing, host resolution, the plugin hook's helpers.
• Integration, under tests/integration: the real pipeline against a mock API with planted quirks (tests/mock-api/server.ts) on a throwaway database; the MCP server over stdio and over HTTP (open, admin token, per-user keys); user requests; contributions; retention; unreachable targets; the Postgres wire path through PGlite; and the plugin hook run as a child process against a real HTTP server.
• npm run mock-api starts the mock API by hand for manual poking.

The Postgres-path test occasionally fails in its setup under the full parallel run and passes alone; re-run before suspecting a change.

Layout

docs/                  spec + stack docs
targets/               one JSON per API under study (parked/ for the ones on hold)
specs/                 hand-authored OpenAPI docs for APIs that publish none
drizzle/               generated SQL migrations (npm run db:generate after editing schema.ts)
src/db/                schema.ts (source of truth), client.ts (PGlite or Postgres), queries.ts,
                       docs.ts (behavior sheets), keys.ts, requests.ts, contributions.ts, retention.ts
src/worker/            pipeline stages: discover, exercise, aggregate/{rules,shape-rules,llm,carry},
                       crossref, verify, write, diff, pipeline.ts, cli.ts; battery/; signature,
                       http, redact, target, requests-runner, contribute
src/llm/               provider adapter (worker only, never the server)
src/schemas/           claim.schema.json
src/server/mcp.ts      MCP server (stdio and Streamable HTTP), instructions block, /contribute
src/dashboard/         Express + static page (local use)
plugin/                Claude Code plugin: PreToolUse hook, .mcp.json, skill, README
.claude-plugin/        marketplace.json (install source for the plugin)
.github/workflows/     test on push/PR; nightly rules-only and weekly LLM refresh
scripts/               export/import, audits, run helpers
tests/                 unit, integration, mock-api, helpers