[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_129":130},"routes/modules.$creator.$moduleId",{"_14":15,"_16":17,"_18":19,"_20":21},"routes/modules.$creator.$moduleId.endpoints",{"_11":12,"_13":-5},"error","Module not found","tools","module",{"_20":22,"_23":24,"_25":26,"_27":28,"_29":30,"_31":32,"_33":34,"_35":26,"_36":37,"_38":-5,"_39":-5,"_40":-5,"_41":42,"_43":44,"_45":46,"_47":48,"_49":50,"_51":52,"_53":54,"_55":-5,"_56":48,"_57":58,"_59":60,"_61":-5,"_62":-5,"_63":-5,"_64":-5,"_65":66,"_67":68,"_69":70,"_71":72,"_73":74,"_75":-5,"_76":-5,"_77":-5,"_78":48,"_79":-5,"_80":81,"_82":83,"_84":85,"_86":87,"_88":89,"_90":91},"apiUrl","https://odel.app/api","creator","rbmuller","moduleId","scherlok","@rbmuller/scherlok","displayName","Scherlok","tagline","Zero-config data-quality monitoring: profile a warehouse, detect anomalies, gate CI.","iconKey","https://avatars.githubusercontent.com/u/44272006?v=4","category","local","status","published","visibility","public","metaDescription","deploymentType","external","workerName","bindingName","mcpConfig","pricingModel","free","creatorUserId","system:external-import","createdAt",1783344535000,"updatedAt",1788164389000,"sourceRegistry","official-mcp-registry","sourceUrl","https://registry.modelcontextprotocol.io/v0/servers?search=io.github.rbmuller%2Fscherlok","sourceVerified",true,"sourceMissingSince","importedAt","upstreamStars",9,"upstreamUpdatedAt",1787753570000,"detectedTransport","detectedAuthType","probeStatus","lastProbedAt","githubRepoUrl","https://github.com/rbmuller/scherlok","githubRepoFullName","rbmuller/scherlok","releaseId","ext-rbmuller-scherlok","version","0.7.0","descriptionMarkdown","\n

\n\n $\"Python$ \n $\"PyPI\"$ \n $\"PyPI$ \n $\"MIT$ \n $\"CI\"$ \n $\"Glama$ \n $\"MCP$ \n\n

\n\n $\"Scherlok\"$ \n\n

Scherlok

\n\n

Your data broke in production. Again.
\nScherlok makes sure it doesn't happen next time.

\n\n

\n\n $\"Scherlok$ \n\n**Zero config. Zero YAML. Zero rules to write.**
\nScherlok learns what \"normal\" looks like, then tells you when something changes.\n\n

\n\n---\n\n## The Problem\n\nEvery data team has the same nightmare:\n\n> A source API silently changes from **dollars to cents**. Revenue dashboards show wrong numbers for **3 weeks** before anyone notices.\n>\n> A column starts returning **NULLs**. A table stops updating. Row counts drop **40% on a Tuesday**. Nobody knows until the CEO asks why the report looks weird.\n\nCurrent tools (Great Expectations, Soda, dbt tests) require you to **define what \"correct\" looks like** before you can detect what's wrong. Hundreds of rules. Dozens of YAML files. And you still miss things — because you can't write rules for problems you haven't imagined yet.\n\n## The Solution\n\nScherlok takes the opposite approach: **learn first, then detect.**\n\n```bash\nscherlok connect postgres://user:pass@host/db # connect once\nscherlok investigate # learn your data\nscherlok watch # detect anomalies\n```\n\nThree commands. Five minutes. Done.\n\n## What It Catches\n\n| Anomaly | What Happened | Severity |\n|---------|---------------|----------|\n| **Volume drop** | Row count dropped 40% overnight | CRITICAL |\n| **Volume spike** | 3x more rows than normal | WARNING |\n| **Freshness alert** | Table hasn't updated in 12h (normally every 2h) | CRITICAL |\n| **Schema drift** | Column removed or type changed | CRITICAL |\n| **NULL surge** | NULL rate jumped from 2% to 45% | WARNING |\n| **Distribution shift** | Column mean shifted 3+ standard deviations (Shewhart-style control limit) | INFO, WARNING above 5σ |\n| **Cardinality explosion** | Status column went from 5 values to 500 | CRITICAL |\n\nEvery anomaly is auto-scored: **INFO**, **WARNING**, or **CRITICAL**. No thresholds to configure.\n\n## Works with dbt\n\nAlready running dbt? Scherlok complements `dbt test` with **automatic** anomaly detection — no rules to write.\n\n```bash\npip install scherlok[dbt]\n\n# After `dbt run`, point Scherlok at your project\nscherlok dbt --project-dir ./my_dbt_project\n```\n\nScherlok reads `target/manifest.json`, discovers every materialized model (`table`, `incremental`, `view`), auto-resolves the connection from your `profiles.yml`, and profiles each model:\n\n```\nInvestigating 4 dbt models in ./my_dbt_project (postgres)\n ✓ stg_customers (12,345 rows)\n ✓ stg_orders (98,765 rows)\n ✗ fct_orders CRITICAL: Row count dropped 42% (98,765 → 57,283)\n ✓ dim_customers_inc (12,300 rows)\n\nSummary: 4 profiled, 1 anomalies (1 critical, 0 warning)\n```\n\nUse it as a CI gate after `dbt run`:\n\n```yaml\n- run: dbt run --target prod\n- run: scherlok dbt --project-dir . --target prod --fail-on critical\n```\n\nOr collapse both steps into one with the wrapper:\n\n```yaml\n- run: scherlok dbt-run-and-watch --project-dir . --target prod --fail-on critical\n```\n\nThe wrapper uses the successful model nodes recorded in `target/run_results.json` after `dbt run`, so partial runs profile only what dbt actually built.\n\nBoth `dbt` and `dbt-run-and-watch` accept `--output json` for CI parsers — a single JSON document on stdout, nothing else.\n\n**Supported adapters:** `postgres`, `bigquery`, `snowflake`, `mysql`, `duckdb`. For others, pass `--connection-string` explicitly.\n\n📖 Full docs: [dbt integration guide →](https://github.com/rbmuller/scherlok/blob/main/src/scherlok/dbt/README.md)\n\n## dbt Package — native tests\n\nPrefer staying inside dbt? Install Scherlok as a dbt package for native data quality tests — no Python CLI needed.\n\n```yaml\n# packages.yml\npackages:\n - package: rbmuller/scherlok\n version: [\">=0.1.0\", \"<1.0.0\"]\n```\n\n```yaml\n# schema.yml\nmodels:\n - name: fct_orders\n tests:\n - scherlok.volume_anomaly:\n sensitivity: 3.0\n - scherlok.row_count_between:\n min_value: 100\n columns:\n - name: email\n tests:\n - scherlok.not_null_proportion:\n max_rate: 0.01\n - name: updated_at\n tests:\n - scherlok.recency:\n days: 2\n```\n\n**Tier 1 — Instant (no setup):** `not_null_proportion`, `row_count_between`, `recency`, `unique_proportion`\n\n**Tier 2 — Auto-learning (Shewhart control limits):** `volume_anomaly`, `null_anomaly` — require the `scherlok_metrics` model to build baseline history.\n\n📖 Full docs: [dbt package README →](https://github.com/rbmuller/scherlok/blob/main/models/_models.yml)\n\n## HTML dashboard\n\n![scherlok dashboard](https://raw.githubusercontent.com/rbmuller/scherlok/main/assets/dashboard-screenshot.png)\n\n```bash\nscherlok dashboard --out report.html\n```\n\nOne self-contained HTML file (~28 KB): KPIs, per-table incidents grouped with first-seen timestamps, `+`/`−`/`~` schema-drift diff, sparklines, and full anomaly history. Auto dark/light theme via `prefers-color-scheme`.\n\n📖 Full docs: [dashboard guide →](https://github.com/rbmuller/scherlok/blob/main/src/scherlok/dashboard/README.md)\n\n## Use it from an AI agent (MCP)\n\nLet Claude Code / Claude Desktop run data-quality checks directly:\n\n```bash\npip install scherlok # scherlok-mcp ships built-in since v0.7.0\n```\n\n```json\n{\n \"mcpServers\": {\n \"scherlok\": {\n \"command\": \"scherlok-mcp\",\n \"env\": { \"SCHERLOK_CONNECTION\": \"postgresql://user:pass@host/db\" }\n }\n }\n}\n```\n\nThe agent gets `list_tables`, `investigate`, `watch`, `status`, `history`, and `check` as tools. Credentials are resolved server-side (never passed by the model), every operation is read-only on the warehouse, and there's no arbitrary-SQL tool.\n\n📖 Full docs: [MCP server guide →](https://github.com/rbmuller/scherlok/blob/main/src/scherlok/mcp/README.md)\n\n## AI-explained alerts (`--explain`)\n\nYour alert says **what** broke. `--explain` adds **why** — and what to check next.\n\n```bash\npip install 'scherlok[explain]'\nexport ANTHROPIC_API_KEY=sk-ant-...\n\nscherlok watch --webhook https://hooks.slack.com/... --explain\n```\n\nWhen anomalies fire, Scherlok makes **one** Claude call for the whole batch and injects a short root-cause hypothesis into the same Slack/Discord/Teams/email/JSON alert:\n\n

\n $\"scherlok$ \n

\n\nWorks on `watch`, `ci`, `check`, `dbt`, and `dbt-run-and-watch`. On dbt projects the hypothesis is **lineage-aware**: upstream parents from `manifest.json` go into the prompt, so cascading failures get traced to the source model instead of alerting on every downstream symptom.\n\n- **What it costs** — one call per fired run (not per anomaly), Claude Haiku 4.5 by default: well under a cent per run (~$0.003). Override the model with `SCHERLOK_EXPLAIN_MODEL`. Runs with zero anomalies make no API call.\n- **What it sends** — aggregates only: the anomaly type/severity/message strings already in your alert, dbt model names, detection timestamps. **Never** warehouse rows, cell values, or credentials — the test suite pins this as a contract.\n- **How to turn it off** — it's opt-in; don't pass `--explain`. If the API call fails (no key, timeout, rate limit), the original alert is delivered unchanged with a one-line note. Alerting never blocks on the LLM.\n\n📖 Full docs: [explainer guide →](https://github.com/rbmuller/scherlok/blob/main/src/scherlok/explainer/README.md)\n\n## How It Works\n\n### 1. `investigate` — Learn the patterns\n\n```bash\n$ scherlok investigate\n\n Profiling 12 tables...\n ✓ users — 45,231 rows, 8 columns\n ✓ orders — 1,203,847 rows, 15 columns\n ✓ products — 892 rows, 12 columns\n ...\n Done. Profiles saved.\n```\n\nScherlok profiles every table: row counts, column types, NULL rates, value distributions, freshness cadence, cardinality. Stores everything locally in SQLite.\n\n### 2. `watch` — Detect anomalies\n\n```bash\n$ scherlok watch\n\n Checking 12 tables against learned profiles...\n\n 🔴 CRITICAL orders volume_drop Row count dropped 52% (1,203,847 → 578,412)\n 🟡 WARNING users null_increase Column \"email\": NULL rate 2.1% → 18.7%\n 🔵 INFO products distribution Column \"price\": mean shifted 3.2σ\n\n 3 anomalies detected. Exit code: 1\n```\n\n### 3. Alert — Slack, CI/CD, or both\n\n```bash\n# Slack\nscherlok watch --webhook https://hooks.slack.com/services/...\n\n# Discord\nscherlok watch --webhook https://discord.com/api/webhooks/...\n\n# Microsoft Teams\nscherlok watch --webhook https://outlook.office.com/webhook/...\n\n# Any endpoint (generic JSON payload)\nscherlok watch --webhook https://my-api.com/alerts\n\n# CI/CD gate (fails pipeline on CRITICAL)\nscherlok watch --exit-code --fail-on critical\n```\n\nAuto-detects Slack, Discord, and Teams from the URL and formats the payload accordingly. Any other URL receives a generic JSON payload.\n\n## CI/CD Integration\n\nUse Scherlok as a data quality gate. The `ci` command does it in one line:\n\n```yaml\n# GitHub Actions\n- name: Data quality check\n run: |\n pip install scherlok\n scherlok config --store s3://my-bucket/scherlok/profiles.db\n scherlok ci ${{ secrets.DATABASE_URL }} \\\n --webhook ${{ secrets.SLACK_WEBHOOK }} \\\n --fail-on critical\n```\n\nIf Scherlok detects a critical anomaly, the pipeline fails. Bad data never reaches production.\n\n## Email alerts\n\n```bash\nexport SCHERLOK_SMTP_HOST=smtp.gmail.com\nexport SCHERLOK_SMTP_USER=alerts@company.com\nexport SCHERLOK_SMTP_PASSWORD=app-specific-password\n\nscherlok watch --email team@company.com --email cto@company.com\n```\n\n## Connectors\n\n```bash\n# PostgreSQL\nscherlok connect postgres://user:pass@host:5432/db\n\n# BigQuery — see src/scherlok/connectors/bigquery.md for auth, billing, CI patterns\npip install scherlok[bigquery]\nscherlok connect bigquery://project-id/dataset-name\n\n# Snowflake\npip install scherlok[snowflake]\nexport SNOWFLAKE_USER=...\nexport SNOWFLAKE_PASSWORD=...\nexport SNOWFLAKE_WAREHOUSE=...\nscherlok connect snowflake://account/database/schema\n\n# MySQL\npip install scherlok[mysql]\nscherlok connect mysql://user:pass@host:3306/dbname\n\n# DuckDB\npip install scherlok[duckdb]\nscherlok connect duckdb:///path/to/file.db\n```\n\n| Database | Status |\n|----------|--------|\n| PostgreSQL | Available |\n| BigQuery | Available |\n| Snowflake | Available |\n| MySQL | Available |\n| DuckDB | Available |\n\n## Remote Storage\n\nShare profiles across CI runs and team members:\n\n```bash\n# AWS S3\nscherlok config --store s3://my-bucket/scherlok/profiles.db\n\n# Google Cloud Storage\nscherlok config --store gs://my-bucket/scherlok/profiles.db\n\n# Azure Blob Storage\nscherlok config --store az://my-container/scherlok/profiles.db\n```\n\n## Why Not [Other Tool]?\n\n| | Great Expectations | Soda | Monte Carlo | **Scherlok** |\n|---|---|---|---|---|\n| Setup time | Hours | 30 min | Weeks | **5 minutes** |\n| Config required | Hundreds of rules | YAML checks | Dashboard setup | **None** |\n| Anomaly detection | Manual thresholds | Paid feature | Yes | **Yes, free** |\n| Self-hosted | Yes | Limited | No (SaaS) | **Yes** |\n| CI/CD gate | Yes | Yes | No | **Yes** |\n| Price | Free | Freemium | $50-200K/yr | **Free, forever** |\n\n## CLI Reference\n\n```\nscherlok connect Connect to a database\nscherlok investigate Profile all tables (learn patterns)\nscherlok watch [-w ] [-e ] Detect anomalies and alert\nscherlok ci [opts] All-in-one CI/CD command (connect + watch + exit code)\nscherlok status Quick health dashboard\nscherlok report Detailed profile summary\nscherlok history [--days N] Timeline of past anomalies\nscherlok config --store Set remote storage\nscherlok version Show version\n```\n\n## Install\n\n```bash\npip install scherlok\n\n# With BigQuery support\npip install scherlok[bigquery]\n```\n\nRequires Python 3.10+.\n\n### Run via Docker\n\nA pre-built image with every warehouse extra (`dbt`, `bigquery`, `snowflake`) is published to GitHub Container Registry on every release tag:\n\n```bash\ndocker run --rm ghcr.io/rbmuller/scherlok:latest version\n```\n\nMount your project directory and inject connection details the same way your CI does it; the entrypoint is the `scherlok` CLI:\n\n```bash\ndocker run --rm \\\n -v \"$PWD:/work\" -w /work \\\n -e SCHERLOK_CONNECTION=postgres://... \\\n ghcr.io/rbmuller/scherlok:latest watch\n```\n\nThe image is built from `python:3.12-slim` and runs unprivileged (`USER scherlok`).\n\n## Contributing\n\nContributions welcome! See [CONTRIBUTING.md](https://github.com/rbmuller/scherlok/blob/main/CONTRIBUTING.md).\n\nWe're especially looking for:\n- New database connectors (Snowflake, MySQL, DuckDB)\n- Anomaly detection improvements\n- Documentation and examples\n\n## License\n\n[MIT](https://github.com/rbmuller/scherlok/blob/main/LICENSE) — Developed by [Robson Bayer Müller](https://github.com/rbmuller)\n","changelogMarkdown","videoKey","requiredSecretsJson","publishedAt","priceCents","trialDays",0,"upstream",{"_109":110,"_111":112,"_113":114,"_115":116,"_117":66,"_118":119},"requiredSecrets",[],"tags",[94,95,96,97,98,99,100,101,102,103,104,105,106,107,108],"assets",[],"stats",{"_92":81,"_93":81},"ratingAvg","reviewCount","anomaly-detection","bigquery","claude","cli","data-engineering","data-observability","data-quality","dbt","duckdb","etl","monitoring","mysql","open-source","postgresql","python","topics",[94,95,96,97,98,99,100,101,102,103,124,125,126,104,105,106,127,107,108,128],"license","MIT","language","Python","registryName","io.github.rbmuller/scherlok","homepage","packages",[120],{"_121":122,"_123":21,"_71":72},"registryType","pypi","identifier","mcp","mcp-server","model-context-protocol","postgres","snowflake","ENV",{"_131":132,"_133":134,"_135":136},"ENVIRONMENT","production","TURNSTILE_SITE_KEY","0x4AAAAAAB8-rEI2gNDf_QhU","SENTRY_DSN","https://45fd1229f78089d5c3ee78be128030e1@o4510583261757440.ingest.de.sentry.io/4510584253448272"]