smart-crawler MCP

E-commerce competitor intelligence, VOC review sentiment, Amazon ASIN analysis, Google Shopping landscape, on-demand URL scraping, and Reddit subreddit growth playbooks — exposed to AI agents as a hosted Model Context Protocol server.

Endpoint: https://smartcrawler.io/mcp/ (streamable-http)
Auth: Authorization: Bearer sck_... — generate a key in the smart-crawler console
Registry: io.github.shulex-voc/smart-crawler

This repository hosts the public MCP manifest for the smart-crawler service. It is the discovery/connection entry point — the service itself is hosted.

Connect

Claude Code

export SMARTCRAWLER_API_KEY=sck_...
claude mcp add --scope user --transport http smart-crawler \
  https://smartcrawler.io/mcp/ \
  --header "Authorization: Bearer $SMARTCRAWLER_API_KEY"

--scope user makes the server visible in every project. Without it, claude mcp add defaults to local scope (only the current directory), a common "I added it but /mcp is empty" pitfall. Verify with claude mcp list or /mcp.

Codex (`~/.codex/config.toml`)

[mcp_servers.smart-crawler]
url = "https://smartcrawler.io/mcp/"
bearer_token_env_var = "SMARTCRAWLER_API_KEY"

Any MCP client

Point your client at https://smartcrawler.io/mcp/ over streamable-http and send your API key as a Bearer token. See mcp.json for the full tool list and server.json for the registry manifest.

Tools (highlights)

Tool	What it does
`reddit_subreddit_playbook`	Full growth playbook for a subreddit's top contributors
`search_competitor_products`	Search competitor catalog by brand / country / keyword / price
`get_voc_reviews`	Consumer VOC reviews from Trustpilot / Reviews.io / Google Maps
`amazon_voc_report`	Amazon ASIN reviews + AI VOC analysis (pain points, listing tips)
`competitor_landscape`	Google Shopping merchant-share ranking for a keyword
`scrape_url`	One-URL scrape with short-lived agent memory
`query_warehouse`	Warehouse-first natural-language query (0 credits)

Full list in mcp.json.

License

MIT

smart crawler