ny-data-registry-mcp

An MCP (Model Context Protocol) server that finds open data across every New York data portal in BetaNYC's NY data portal registry: which portal covers a place or topic, how to query it, and, with the harvested index, which datasets and columns exist and in what formats.

It complements the socrata MCP rather than replacing it. This server knows where data is (140 sources across state, county, city, agency, library and nonprofit portals) and hands the agent the right Socrata domain or ArcGIS service. Querying the data stays with socrata and the other BetaNYC MCPs.

Built by BetaNYC. Status: 0.1.0. Package: @betanyc/ny-data-registry-mcp.

What it does

Nine tools, in two groups.

Registry tools (work out of the box, no index needed):

Index tools (need the harvested SQLite index):

All tool inputs are strict: an unknown parameter is rejected by name, never silently dropped (same rule as nyc-311-mcp).

Questions to try

Ask your AI assistant these once the server is connected. Where we ran a question on 2026-11-05, what it found is noted. Portals change, so your results may differ.

Which datasets match my words? (search_datasets)

• "Which New York cities publish housing code violation data? Search for 'housing code violations' and 'code enforcement', and list the results by city." Finds New York City (HPD violations, NYCHA violations, tenant complaints, DOB violations), Buffalo (code violations) and Syracuse (code violations, unfit properties, rental registry).
• "Do any New York cities publish zoning data? If so, what formats do they publish them in?" Finds New York City, Buffalo, Rochester, Glens Falls and Cortland, plus zoning layers on the Westchester, Albany, Dutchess, Tioga, Wayne, Oswego and Warren county portals.
• "Find flood zones, as GeoJSON." Finds the statewide FEMA flood hazard layer, plus Buffalo and Cortland County.
• "Find restaurant inspection data." Top hit: NYC's DOHMH restaurant inspection results.
• "Where are the farmers markets?" Finds the statewide list on data.ny.gov and Albany County's.
• "Find EV charging stations, as a CSV file." Finds two NYC charging datasets and the statewide alternative fuel stations list.
• "Housing code violations in Syracuse." / "Bike lanes in Buffalo." / "Street trees in Buffalo." Narrow a search to one place.
• "Where are the polling places?" Finds Suffolk and Tompkins counties.
• "Crash data updated since 2026-10-01." Use a date to see only datasets that changed recently.

Who has data, and for where? (find_sources, list_sources_for_place)

• "Which New York portals have health data?"
• "Which state agencies have a real API?" / "Which portals need an API key?"
• "What open data does Erie County have?" Erie County was checked and has no portal; the answer says so.
• "What can I get for Syracuse?" / "What covers Queens, or the North Country?"

How do I use it? (how_to_query, get_dataset_columns, find_joinable)

• "How do I pull data from the NYS Department of Health portal?"
• "Does Westchester County publish zoning data, and what columns does it have?"
• "What columns does Buffalo's code violations dataset have?"
• "What other datasets can I join to NYC's housing violations by BBL?"

Is it current? (whats_new, check_source, registry_info)

• "What datasets were added or changed since October 1?"
• "Is Albany's data portal still working?"
• "How many sources do you track, and when was the list last updated?"

Tips

• Try more than one wording. Cities name the same thing differently: "code enforcement" finds datasets that "housing code violations" misses.
• New York City crowds the results. NYC's matches tend to fill the top of the list. To see the rest of the state, name a place or ask a second time for results outside New York City.
• Add a date to skip old data. "School enrollment" alone returns NYC tables from 2010–2011; add a date, such as "updated since 2025-01-01", to skip them.
• An empty result can still have data elsewhere. The index covers Socrata and ArcGIS portals only. A city may publish on a site this server does not search.
• Some maps are not official. The Westchester and Tioga county zoning layers say they are not the legal boundaries. Check the town's own map before relying on one.
• It finds the data; another tool fetches it. Hand a Socrata dataset to the socrata MCP. For an ArcGIS layer, open or download it from the portal.

Install

Requires Node 22.16 or later (it uses Node's built-in node:sqlite, so there is nothing native to compile; 22.16 is the first official Node whose SQLite includes FTS5 full-text search).

Add it to your MCP client's config (Claude Desktop, Claude Code, Cursor, VS Code and others use this shape):

"ny-data-registry-mcp": {
  "command": "npx",
  "args": ["-y", "@betanyc/ny-data-registry-mcp"]
}

That's all. The first time it starts, it downloads the dataset index by itself (about 19 MB, unzipped to about 110 MB). This takes a few seconds on a normal connection. The five registry tools work right away. If you ask a dataset question while the download is still running, the tool waits up to 45 seconds and then says "try again in a minute".

From source instead:

git clone https://github.com/BetaNYC/ny-data-registry-mcp && cd ny-data-registry-mcp
npm install            # also builds
npm test               # offline; no network

and point your client at node /path/to/ny-data-registry-mcp/dist/index.js.

The dataset index

The four index tools (search_datasets, get_dataset_columns, find_joinable, whats_new) read a prebuilt SQLite index. It holds metadata only: titles, descriptions, column names and types, formats and links. No dataset contents.

• Where it comes from: the index-latest release, at a fixed URL: https://github.com/BetaNYC/ny-data-registry-mcp/releases/download/index-latest/ny-data-index.sqlite.gz. Dated snapshots are kept as index-YYYY-MM-DD releases.

• Checked before use: the server checks both SHA-256 sums in the release's SHA256SUMS file (the .gz and the unzipped .sqlite) and the schema version. A file that fails any check is never installed.

• Where it goes: ${HOME}/.cache/betanyc/ny-data-registry/ny-data-index.sqlite (change with NY_DATA_INDEX).

• When: only when the file is missing. An existing index is never replaced on its own.

• Get or refresh it by hand:

  bashCopy

  npx -y -p @betanyc/ny-data-registry-mcp ny-data-registry download          # if missing
  npx -y -p @betanyc/ny-data-registry-mcp ny-data-registry download --force  # replace with the newest
  

• Turn the download off: set NY_DATA_INDEX_AUTO_DOWNLOAD=0. Then the dataset tools reply with these steps instead. Or build your own index with ny-data-registry harvest (see "Refresh").

Environment variables (all optional)

Data sources

• The registry. BetaNYC/grounding-ai-with-ny-open-data resources/ny-data-portal-registry.json (140 rows). A copy is bundled in data/. On startup the server makes a conditional request (ETag) for the published copy on GitHub Pages, caches it under ${HOME}/.cache/betanyc/, and falls back to the bundled copy on any failure. This server never edits the registry.
• Human-editable tables in data/:
  • platform-families.json: maps each of the registry's 101 free-text platform strings to one family (socrata, arcgis-hub, arcgis-enterprise, arcgis-server, ckan, geoblacklight, custom-api, none). A string not in the table is reported as unmapped, never guessed; 7 strings are deliberately null because the registry marks them UNCONFIRMED.
  • ny-places.json: 62 counties, 62 cities, 5 boroughs, 8 regions.
  • key-columns.json: the curated join keys.
  • handoffs.json: which BetaNYC MCP serves which registry row.
  • harvest-config.json: skips (ANHD, Rockland, Columbia GeoBlacklight) and caps.
  • synonyms.json: a short civic synonym list ("property" → parcels, tax lots, BBL).
• The harvested index, built from the portals themselves (next section).

The harvest

node dist/cli.js harvest reads every live registry row and harvests the ones a documented catalog API covers:

Everything else (custom APIs, CKAN, GeoBlacklight, standalone ArcGIS Server directories, pages with no API) is described by the registry tools but not harvested in this version.

Guards

• Esri's own content is excluded (owner esri…, org P3ePLMYs2RVChkJx). For Enterprise portals and ArcGIS Online orgs, items from another org are not counted under that portal; this is what turns Monroe's 108 search hits into its own content. Hub sites are not filtered by org, because a Hub's catalog is curated by its owner and aggregators like the State GIS Clearinghouse deliberately list other agencies' layers.
• Out-of-state guard: an item whose extent lies wholly outside New York is dropped; a portal whose items are mostly outside New York is refused and purged. Placeholder extents ([0,0,0,0]) and projected coordinates (State Plane) count as unknown, never as outside.
• Stable paging: Hub pages are requested sorted by id. A listing that comes back with fewer unique items than the portal's own total is treated as incomplete, so nothing is marked removed from a shaky listing.
• Politeness: one descriptive User-Agent with no email address; at least 1 s between requests to a host; robots.txt Disallow and Crawl-delay honored (every Hub site asks for 60 s, so Hub listing is slow on purpose); conditional requests by ETag; one bounded retry on 429; a per-portal cap on layer requests. A bot block is reported, never worked around.
• Never called: rows needing a key, ANHD, Rockland County, undocumented endpoints.

The index stores metadata only (titles, descriptions, field names, links). It never downloads data.

Incremental: datasets carry first_seen, last_seen and a content hash. A layer whose modified date has not changed keeps its stored columns and is not re-requested. A dataset is marked removed only after a complete harvest of its portal.

Refresh

• Registry: picked up on next server start (conditional GET of the Pages copy).
• Index: re-run node dist/cli.js harvest. .github/workflows/harvest.yml is written to do this nightly in GitHub Actions and publish the SQLite file as a release asset, but it is not enabled (schedule commented out). Index releases are published by hand for now; the first is index-2026-11-05.

How it was checked

• npm test: offline tests from recorded fixtures (test/fixtures/, each says which request produced it), including a stdio smoke test that starts the server and calls all nine tools.
• node dist/cli.js eval: a fixed set of real questions (eval/questions.json) scored against the real index. Results: eval/runs/.

Latest results (2026-11-05, office-mac), all in eval/runs/:

• npm test: 44 / 44 pass.
• Question set: 27 / 27 in the top 3, 0 invariant violations. The set was written by the same agent that built the tool, so treat 100% as an upper bound; two checks (Q01, Q13) were tightened during the build when a loose check passed on the wrong county's parcels.
• Index: 9,569 datasets, 185,352 columns (5,729 with a join key) from 35 of 36 harvestable portals; 110 MB SQLite (about 18 MB gzipped). Full harvest: 924 s and 3,694 requests; an ArcGIS-only re-run: 878 s and 1,403 requests (unchanged layers are not re-requested).
• 311 (erm2-nwe9) joins: BBL 426, borough 1,073, community district 510, council district 426 datasets (Scout, name equality only: 331 / 512 / 286).

Limits

• Coverage is the registry's. A place not in the registry has not been checked; the tools say so.
• Not harvested yet: standalone ArcGIS Server REST directories (NYS DOT, ITS, many counties), GeoBlacklight (Cornell), CKAN (catalog.data.gov), custom APIs.
• Row counts are not collected (Socrata Discovery does not return them; ArcGIS would need one more request per layer). downloads and update_frequency exist only for Socrata.
• Columns for ArcGIS are capped at 250 layer requests per portal per run; the rest show columns_status: not-fetched and fill in on later runs.
• Search is keyword search (SQLite FTS5 with a small synonym list), not semantic search.
• Join keys are by column name only; values are not compared.
• Orange County's portal is unreachable from Node because its server does not send its intermediate TLS certificate. Certificate checks are not disabled.
• Syracuse's Hub returned 126 unique items of 136 on one run even with sorted paging; the guard kept the index whole, but a short listing can still miss new items until the next run.
• The index is about 110 MB uncompressed, larger than the scoping estimate (low tens of MB), mostly Socrata column metadata and the full-text index.

Terms of use

Read before any adapter was written; details and quotes in docs/terms.md.

• Socrata / Tyler: the Discovery API's stated purpose is "automated searching, research, and exploration"; read-only access needs no authentication; an app token is recommended. Per-portal terms pages were not found (not verified).
• Esri: ArcGIS Online public content is for personal or noncommercial use under each item's own terms, and owners who share publicly grant end users permission to use and reproduce it. The Master Agreement's "may not scrape, download, or store Data" clause defines Data as Esri's own commercial datasets, which are excluded here. Hub robots.txt asks for a 60 s crawl delay, which is honored.

Related

• Scout field mapping: docs/scout-mapping.md
• Proposed upstream registry change (platform_family): docs/upstream/

License

MIT