Hello world

\", 8000);\r\nprintln!(\"{}\", result.text);\r\n\r\n// Pass 0 to disable the per-page cap entirely (no truncation)\r\nlet full = clean(\"

Hello world

\", 0);\r\nassert!(!full.truncated);\r\n\r\n// Fetch and clean a single page\r\nlet config = Config::default();\r\nlet page = fetch(\"https://example.com\", &config).await?;\r\n\r\n// Full search pipeline\r\nlet results = query(&[\"rust async programming\"], &config).await?;\r\nfor source in &results.sources {\r\n println!(\"[{}] {} — {} chars\", source.id, source.title, source.content.len());\r\n}\r\n\r\n// Backend partial failures (CAPTCHA, rate-limits, engine outages) are surfaced\r\n// in `warnings` rather than as Err. Empty sources + non-empty warnings means\r\n// \"all backends blocked\" — distinct from a legitimate \"no matches found\".\r\nif results.sources.is_empty() && !results.warnings.is_empty() {\r\n eprintln!(\"all backends failed: {:?}\", results.warnings);\r\n}\r\n```\r\n\r\n### Text-map: rewrite HTML content without breaking markup\r\n\r\nExtract text nodes, manipulate them (translate, rewrite, simplify), and rebuild\r\nthe HTML with structure, attributes, and links intact.\r\n\r\n```rust\r\nuse webshift::{extract_text_nodes, replace_text_nodes, TextReplacement};\r\n\r\nlet html = r#\"

Hello world

\"#;\r\nlet map = extract_text_nodes(html);\r\n// map.nodes = [(0, \"Hello\"), (1, \"world\")]\r\n\r\nlet replacements = vec![\r\n TextReplacement { id: 0, text: \"Ciao\".into() },\r\n TextReplacement { id: 1, text: \"mondo\".into() },\r\n];\r\nlet result = replace_text_nodes(html, &replacements).unwrap();\r\n// →

Ciao mondo

\r\n// href untouched, tags intact, only text changed.\r\n```\r\n\r\nRequires `features = [\"text-map\"]`. See [Use Cases #11](https://github.com/x-hannibal/webshift/blob/main/docs/USE_CASES.md#11-translate-html-without-breaking-layout-text-map) for a full translation example.\r\n\r\n### Feature flags\r\n\r\n| Feature | Default | Enables |\r\n|---------|---------|---------|\r\n| `backends` | on | All search backends + query pipeline |\r\n| `llm` | off | LLM client, expander, summarizer, LLM reranking |\r\n| `text-map` | off | `extract_text_nodes()` + `replace_text_nodes()` — DOM round-trip for content rewriting |\r\n\r\n---\r\n\r\n## Integrations\r\n\r\n| Platform | Guide |\r\n|----------|-------|\r\n| Claude Desktop, Claude Code, Cursor, Windsurf, VS Code | [IDE Integration](https://github.com/x-hannibal/webshift/blob/main/docs/integrations/IDE.md) |\r\n| **Zed** — native extension with auto-download and Configure Server modal | [Zed Extension](https://github.com/x-hannibal/webshift/blob/main/docs/integrations/ZED_EXTENSION.md) |\r\n| Gemini CLI, Claude CLI, custom agents | [Agent Integration](https://github.com/x-hannibal/webshift/blob/main/docs/integrations/AGENT.md) |\r\n\r\n---\r\n\r\n## Beta Status\r\n\r\nWebShift is in **beta**. Core functionality is stable and the server is used daily,\r\nbut the API surface may still change before 1.0.\r\n\r\n**Feedback is very welcome.** If something doesn't work as expected, behaves oddly,\r\nor you have a use case that isn't covered:\r\n\r\n> [Open an issue on GitHub](https://github.com/x-hannibal/webshift/issues)\r\n\r\nBug reports, configuration questions, and feature requests all help shape the roadmap.\r\n\r\n## Contributing\r\n\r\nContributions are welcome! Please see [CONTRIBUTING.md](https://github.com/x-hannibal/webshift/blob/main/CONTRIBUTING.md) for detailed guidelines on:\r\n- Development setup and workflow\r\n- Code style and conventions\r\n- Testing requirements\r\n- Documentation standards\r\n- Pull request process\r\n\r\n## License\r\n\r\nMIT License — see [LICENSE](https://github.com/x-hannibal/webshift/blob/main/LICENSE) for details.\r\n\r\n## Links\r\n\r\n- **[GitHub Repository](https://github.com/x-hannibal/webshift)** — Source code and issues\r\n- **[Docs.rs](https://docs.rs/webshift)** — API documentation \r\n- **[MCP Registry](https://registry.modelcontextprotocol.io/?q=mcp-webshift&all=1)** — WebShift on Model Context Protocol Registry\r\n- **[MCP Protocol](https://modelcontextprotocol.io/specification/2025-11-25)** — Model Context Protocol specification\r\n\r\n---\r\n\r\n**Need help?** Check the [documentation](https://github.com/x-hannibal/webshift/tree/main/docs) or open an [issue](https://github.com/x-hannibal/webshift/issues) on GitHub.\r\n\r\n\r\n","changelogMarkdown","videoKey","requiredSecretsJson","publishedAt","priceCents","trialDays",0,"upstream",{"_108":109,"_110":111,"_112":113,"_114":115,"_116":-5,"_117":118},"requiredSecrets",[],"tags",[93,94,95,96,97,98,99,100,101,102,103,104,105,106,107],"assets",[],"stats",{"_91":80,"_92":80},"ratingAvg","reviewCount","ai-agents","bing-search","bm25","brave-search","context-window","denoiser","exa","google-search","html-cleaner","html-to-text","llm","rust","searxng","serpapi","tavily","topics",[93,94,95,96,97,98,99,100,101,102,103,119,120,104,105,106,107,121,122,123],"license","MIT","language","Rust","registryName","io.github.x-hannibal/webshift","homepage","packages",[],"mcp","mcp-server","text-extraction","web-scraping","web-search","ENV",{"_126":127,"_128":129,"_130":131},"ENVIRONMENT","production","TURNSTILE_SITE_KEY","0x4AAAAAAB8-rEI2gNDf_QhU","SENTRY_DSN","https://45fd1229f78089d5c3ee78be128030e1@o4510583261757440.ingest.de.sentry.io/4510584253448272"]