Octocrawl — Web-to-LLM Context Extraction

Website · Docs · Connect MCP · Blog · Changelog

Octocrawl turns web pages into Markdown and data for RAG and agent workflows. It records how it got each page, so every result can be checked.

When a page fails, Octocrawl says so and says why. It does not report a blocked or empty page as a success.

Try it without installing anything at octocrawl.dev, or read the documentation.

The web preview reads one page at a time, five a day. Besides Markdown it returns the page's links and metadata, and up to 20 fields read without a model (how it is deployed).

Quick start

Connect your agent to hosted Octocrawl. No key is needed to start (steps for Claude Code, Cursor, OpenCode and Codex):

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

Or run it on your own computer. You need Node.js 22.13+ or 24+:

npx octocrawl scrape https://example.com --markdown
pip install octocrawl-client      # the Python client of a running API (npx octocrawl serve)
npm install @octocrawl/sdk        # the TypeScript client; @octocrawl/mcp is the MCP server

To give your agent every tool, with no daily limit, run the API locally and connect the MCP server to it:

npx octocrawl serve                                  # keep it running: the API on 127.0.0.1:8787
claude mcp add octocrawl -- npx -y @octocrawl/mcp    # the published stdio server, a client of that API

The hosted service offers scrape and map. The local server adds crawl, batch, the Amazon.sg product tool and the Monitor tools. Running Octocrawl covers both, plus hosting a server for others.

Why this exists

Most crawlers report "success" when they return empty pages, challenge screens or the wrong content. Octocrawl makes failure visible and fixable:

Before (typical crawler):

✓ Fetched example.com/article
  Status: 200 OK
  Content: 953 bytes

After (Octocrawl):

✗ Fetched example.com/article
  Status: blocked (cloudflare_challenge)
  Lane: http → escalated to browser_local
   Evidence: artifacts=[] (no screenshot or DOM snapshot was produced)
  Cost: 847 tokens, 2.3s, $0.0042
  Fix: needs user login or proxy (tier 1b/2)

What's different

1. Failure is a result. A page is empty_verified, blocked or failed with a reason, never a silent empty page. An error page keeps its status and Markdown as evidence, not as a success. Details.
2. Five false-success checks. Octocrawl checks for challenge text, wrong-page content, missing facts, truncation and too little content.
3. Execution ladder. A request moves from HTTP to a browser, then to your login or a proxy, as needed. Each attempt is traced, and the task's cost is counted.
4. Ground-truth benchmark. A 56-case fixture suite (soft 404s, challenge pages, SPAs, timeouts, zip bombs, tables) measures the false-success rate.
5. Honest evidence. An unknown value is null, not zero. artifacts: [] means no screenshot or DOM snapshot was made. bytesWire: null means wire bytes were not measured.
6. One Evidence Record. Every scrape result, batch item and crawl page carries an evidenceRecord. It holds the final URL, redirects, fetch time, status, lane, robots.txt decision, hashes and field evidence. A versioned JSON Schema describes it; see the reference.

Documentation

The site's docs at octocrawl.dev/docs cover MCP setup for four clients, task guides and limits. They are built from apps/public-web/content.

The full reference lives in docs/:

Walkthroughs of real runs are on the blog: Claude Code web scraping with MCP, Web scraping for RAG, Find all pages on a website, Scrape a website with login, in your own Chrome, Extract structured data without an LLM, Scrape a list of URLs and keep every failure.

For researchers, two guides walk through a real run: From a URL list to a CSV with evidence and Citing web data in a paper.

For the Monitor → result → HTTPS event → restart workflow, follow the onboarding guide.

Work on the code

You need Node.js 22.13+ or 24+ and npm. The PDF engine, pdf.js, needs 22.13 or later.

git clone https://github.com/77777R7/Octocrawl.git
cd Octocrawl
npm ci
npx playwright install chromium
npm run typecheck
npm test
npm run scrape -- https://example.com --markdown
npm run crawl -- https://example.com --max-pages 20

The packages are published from this repository. Each week the Install check workflow runs the npx line from an empty cache on macOS, Windows and Linux.

Names: Octocrawl and w2l

The product is called Octocrawl. Its earlier working name was W2L (Web-to-LLM).

Some identifiers still use the old name. They stay until a change renames them on purpose, because renaming them breaks existing setups and records:

• environment variables such as W2L_API_PORT and W2L_TASK_ROOT;
• the data directory .w2l/;
• the internal workspace packages @w2l/* and the checkout's w2l command (npm run w2l);
• the SDK classes W2L and W2LError;
• the Evidence Record version w2l.evidence/1;
• the research identity w2l-research, which sites' robots.txt can name.

The published packages already use the new name: octocrawl, @octocrawl/* and octocrawl-client. The full list is in AGENTS.md.

Benchmark

Run the fixture suite against the bare HTTP baseline:

npm run bench

The bare HTTP baseline has a high false-success rate on purpose. It has no content extraction, challenge detection or redirect handling. Benchmarks shows the expected output and the throughput and crash-recovery run.

Repository structure

packages/
  contracts/       TypeScript types and ground-truth schema (MIT)
  fixtures/        HTTP server with 56 ground-truth test cases
  http-core/       robots.txt parser (ReDoS-resistant)
  extract-tf/      Content extraction, HTML → Markdown, PDF text
  runtime/         TaskStore, frontier, bounded crawl orchestrator
  bench/           Benchmark runner, ladder CLI (w2l-ladder), scoring
  cli/             octocrawl: scrape, crawl, batch, map, serve (AGPL)
  api/             REST server (AGPL)
  sdk/             TypeScript client (MIT)
  mcp/             stdio and restricted Streamable HTTP MCP server (AGPL)
  public-preview/  Server for the public preview at octocrawl.dev

apps/public-web/   The octocrawl.dev site and its docs pages
python/            Python client octocrawl-client (MIT)
examples/          Runnable Monitor + Delivery workflow and webhook receiver
docs/              Reference, guides, ADRs and dated records (index: docs/README.md)
ROADMAP.md         The current phase and what is paused
CHANGELOG.md       Release notes

Roadmap

ROADMAP.md starts with a summary of the current phase. The earlier milestone checklist is in docs/roadmap/milestones.md.

Contributing

We use the Developer Certificate of Origin (DCO) instead of a CLA. Every commit needs a Signed-off-by line:

git commit -s -m "Your commit message"

See CONTRIBUTING.md for details, including how to write docs.

License

Server-side code, the CLI (@octocrawl/cli, octocrawl) and the MCP server (@octocrawl/mcp): AGPL-3.0
Client libraries: MIT: the TypeScript SDK (@octocrawl/sdk, which includes the workspace's @w2l/contracts, also MIT) and the Python client (octocrawl-client)

See PHASE1_ENGINEERING_NOTES.md §1.3 for the rationale.

Why AGPL?

AGPL requires network-deployed modifications to remain open. Anyone can fork, modify, and host Octocrawl — as long as they share those modifications. The real differentiator is the name (trademark) and the hosted service, not the license lock.