Iris — stop shipping agents on vibes

Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account. Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 20 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds, and the agent's own tool calls — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry; your traces never leave your machine.

Requires Node.js 20 or later. Check with node --version.

A failure on screen in 60 seconds

No agent wiring, no config — one command:

npx @iris-eval/mcp-server --demo

This seeds a demo database — a handful of small agents with a week of runs — and serves the dashboard against it at http://localhost:6920 (your browser opens automatically on first run). The dashboard lands on Failures: what failed, worst and newest first. Worth clicking into — a PII leak caught by the safety rules, a flagged prompt-injection attempt, and a failed LLM-judge score with its rationale.

Demo data lives in its own database (demo.db in your Iris home directory — ~/.iris on macOS/Linux, %USERPROFILE%\.iris on Windows) and never mixes with your real traces. Remove all of it with one command:

npx @iris-eval/mcp-server --demo-clear

Hook up your own agent

Add Iris to your MCP config. Works with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI — and any other MCP-compatible agent. One block, dashboard included:

{
  "mcpServers": {
    "iris-eval": {
      "command": "npx",
      "args": ["@iris-eval/mcp-server", "--dashboard"]
    }
  }
}

Your agent discovers Iris's nine tools on connect, and the dashboard serves at http://localhost:6920. Now paste this to your agent:

Log that last task to Iris and evaluate the output.

The trace lands on the dashboard with its scores. Prefer the MCP server headless? Drop --dashboard from the args — you can open the same dashboard any time with npx @iris-eval/mcp-server --dashboard.

One thing worth knowing up front: MCP tools are called when the model decides to call them. Iris doesn't intercept your agent, so traces are logged when your agent asks it to log them — either because you told it to, or because your code calls the tools directly. Ask your agent to "log this to Iris and evaluate it" and it will. If you want capture that doesn't depend on the model choosing, POST /api/v1/traces does exactly that — your code sends the trace over plain HTTP, no model in the loop (see docs/http-ingest.md). The CLI and SDKs on the roadmap will be thin clients over the same endpoint.

Capture over HTTP (no model in the loop)

The ingest endpoint lives on the dashboard port — 6920 by default, not the MCP transport port — and it exists only while the dashboard is running. Pass --dashboard (or set IRIS_DASHBOARD=true); --transport http on its own does not start it, and a request to the transport port returns 404. With the dashboard up, anything that can send an HTTP request can log a trace — and optionally run the deterministic evals in the same request. GET /api/v1/capabilities on the same port says what this server can judge, what each rule needs, the judge state with the steps that enable it, and the limits — the same object the MCP resource iris://capabilities serves — so an HTTP caller has the frame an MCP client gets at initialize:

curl -s -X POST "http://127.0.0.1:6920/api/v1/traces" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "support-bot",
    "input": "What is the refund policy?",
    "output": "Refunds are available within 30 days of purchase.",
    "evaluate": true,
    "eval_type": "safety"
  }'

Returns 201 with the stored trace_id and the evaluation result (in --demo mode the endpoint refuses writes with 403, so demo data never mixes with yours). The endpoint accepts the same body as the log_trace tool and sits behind the same middleware stack as the rest of the dashboard: loopback bind and the DNS-rebinding guard by default, plus Bearer auth when you set one. Two plain facts about it: it accepts unauthenticated writes unless Iris was started with --api-key (or IRIS_API_KEY) — the loopback bind is what keeps it to your machine by default, so set a key before binding beyond loopback; and what it stores is verbatim — input and output land in iris.db exactly as sent, including any text no_pii goes on to flag. Full contract, field reference, and error semantics: docs/http-ingest.md.

Verify your install

npx @iris-eval/mcp-server --self-test   # offline diagnostic; exit 0 = healthy, 1 = a check failed
npx @iris-eval/mcp-server --version     # prints the bare version, e.g. 0.5.1

--self-test first creates your Iris home if it is missing and checks that it is writable (exit 1, naming the path, if it is not), then runs its checks — storage round-trip, a planted SSN and a planted injection caught by the safety rules, dashboard boot, the DNS-rebinding guard — inside an isolated temp home, so your real database is never opened. Everything Iris writes lives under one directory, your Iris home: ~/.iris by default (%USERPROFILE%\.iris on Windows), or wherever IRIS_HOME points. That is where iris.db, config.json, custom-rules.json, audit.log, preferences.json and the demo files live; point IRIS_HOME at a scratch directory to try Iris without touching your real data.

claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server

{
  "servers": {
    "iris-eval": {
      "command": "npx",
      "args": ["@iris-eval/mcp-server"]
    }
  }
}

{
  "context_servers": {
    "iris-eval": {
      "command": {
        "path": "npx",
        "args": ["@iris-eval/mcp-server"]
      }
    }
  }
}

[mcp_servers.iris-eval]
command = "npx"
args = ["@iris-eval/mcp-server"]

Other Install Methods

# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-mcp --dashboard

# Docker — two servers, two ports: 3000 = MCP HTTP transport,
# 6920 = dashboard (which also serves the POST /api/v1/traces ingest endpoint)
docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data ghcr.io/iris-eval/mcp-server

Tip: Global install (npm install -g) stores traces persistently at ~/.iris/iris.db. With npx, traces persist in the same location, but startup is slower due to package resolution.

What You Get

Where this is going next: the roadmap.

Measured, not claimed

Every built-in rule has a published precision, recall and F1 with 95% confidence intervals, measured on a labelled corpus that lives in this repository (proof/corpus/) and regenerates with one command — npm run proof — offline, with no key and no model in the loop. CI re-runs the measurement on every pull request and fails if the committed numbers differ from what the code produces, so a rule cannot change without its numbers changing with it. The numbers are on iris-eval.com/proof and in proof/RESULTS.md; how the corpus was made, what it is not, and how to read an interval are in docs/proof.md. The corpus is synthetic and model-labelled — a human blind label is pending, and the page says so; node proof/blind-sample.mjs draws the reproducible sample that will settle it.

MCP Tools

Iris registers nine tools that any MCP-compatible agent can invoke — full rule + trace lifecycle + LLM-as-judge + semantic citation verification:

• log_trace — Log an agent execution with spans, tool calls, token usage, and cost
• evaluate_output — Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)
• get_traces — Query stored traces with filtering, pagination, and time-range support
• list_rules — Enumerate deployed custom eval rules (read-only)
• deploy_rule — Register a new custom eval rule so it fires on every evaluate_output of that category
• delete_rule — Remove a deployed custom rule (destructive, idempotent)
• delete_trace — Remove a single stored trace by ID (destructive, tenant-scoped)
• evaluate_with_llm_judge — Semantic eval via LLM (Anthropic or OpenAI). Five templates: accuracy, helpfulness, safety, correctness, faithfulness. Cost-capped, per-eval pricing disclosed. Bring your own API key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY) — Iris doesn't proxy or relay LLM calls.
• verify_citations — Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement as evaluate_with_llm_judge.

Enable the LLM judge (optional; the deterministic rules never need it)

1. Get an API key from Anthropic or OpenAI.
2. Put it in the environment of the process that runs Iris, not only your shell. Claude Code, Claude Desktop, Cursor and most MCP clients: the "env" block of the iris-eval entry in your MCP config — "iris-eval": { "command": "npx", "args": ["-y", "@iris-eval/mcp-server"], "env": { "IRIS_ANTHROPIC_API_KEY": "sk-ant-..." } } (IRIS_OPENAI_API_KEY for an OpenAI key). Docker: -e IRIS_ANTHROPIC_API_KEY=... on the run command. HTTP or CI: export it before starting iris-mcp.
3. Restart the MCP session. A running process never sees a variable set after it started.
4. Confirm from inside your client: read iris://capabilities — judge.enabled must be true there. A key exported in your shell is not passed to the process your client spawns unless its config lists it. On a machine, npx @iris-eval/mcp-server --self-test prints the judge line for that shell, and GET /api/v1/health reports judge.enabled on a running dashboard.
5. Spend guard: each call is capped by IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL (default 0.25 USD) and refused before any spend if the worst case would exceed it. Iris calls the provider directly with your key and never proxies it.

When IRIS_OTEL_ENDPOINT is configured, log_trace calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See docs/otel-integration.md.

How passed is decided

evaluate_output returns both a score and a passed flag — they answer different questions:

• score (0..1) is the weighted average across the rules that ran — a quality gradient.
• passed is the ship/no-ship verdict: true only when the score clears the pass threshold (default 0.7) and no critical rule failed.

Genuine safety violations hard-fail. By default no_pii, no_injection_patterns, and no_blocklist_words are critical rules: if one fails, the eval reports passed: false no matter how well the other rules scored, and the response names the culprits in critical_failures. A leaked SSN can't be averaged away. Which built-in rules are critical is a deployment setting (eval.criticalRules / eval.nonCriticalRules); every rule result carries the effective critical flag and criticalSource, and list_rules reports the roster this server applies. Custom rules deployed with severity: "high" or "critical" hard-fail the same way; low/medium severities only affect the score. One boundary to know, stated the same way on every surface: a critical rule that skipped (missing context, a broken definition, or a regex killed at the sandbox budget) has not judged the output and does not veto — every such rule is named in critical_skipped. A gate that must fail closed treats a non-empty critical_skipped as unknown, not clean, and may treat any budgetExceeded skip in rule_results the same way.

For CI gates: if you omit eval_type, every bundle runs — completeness, relevance, safety, cost and any custom rules — and the response says eval_type: "all" with a note that the default ran, plus a per-bundle categories map. A bundle with nothing to judge (cost without cost_usd, relevance without input) reports passed: null there — not evaluated, not failing — and never counts toward the verdict. The response always echoes the eval_type that ran, so your gate can verify coverage; key on passed for the verdict and name a bundle only when you want a narrower run.

Authoring a custom rule

Two ways to add a rule. Inline rules ride along on one evaluate_output call (custom_rules, up to 10 per call); they fire alongside whatever eval_type bundle you chose, or alone with eval_type: "custom". Deployed rules are registered once with deploy_rule, persist in custom-rules.json under your Iris home, and fire on every future evaluate_output of their evalType. The definition is the same shape either way:

deploy_rule wraps the definition with name, an optional description, evalType (completeness · relevance · safety · cost · custom) and severity. Severity says what a failure means: low/medium only lower the score; high/critical hard-fail the evaluation — passed: false, the rule named in critical_failures — whatever the weighted score says. A rule that skips (a cost_threshold rule with no cost_usd, or a regex killed at the 100 ms sandbox budget) has not judged the output and is listed in critical_skipped instead. Deploy a critical rule that forbids internal hostnames in anything the agent says:

{
  "name": "no_internal_hostnames",
  "description": "Output must not mention internal hostnames.",
  "evalType": "safety",
  "severity": "critical",
  "definition": {
    "name": "no_internal_hostnames",
    "type": "regex_no_match",
    "config": { "pattern": "\\b[a-z0-9-]+\\.internal\\.example\\b", "flags": "i" }
  }
}

The response is the persisted rule — keep the id for delete_rule:

{ "rule": { "id": "rule-588823d0", "name": "no_internal_hostnames", "evalType": "safety", "severity": "critical", "enabled": true, "version": 1, "definition": { "…": "…" } } }

From the very next evaluate_output with eval_type: "safety", an output that mentions db-primary.internal.example comes back passed: false with critical_failures: ["no_internal_hostnames"] — even though all five built-in safety rules passed and the weighted score is 0.895. Regex patterns must pass a ReDoS check at deploy time and always run in a sandbox worker under a hard 100 ms deadline. list_rules shows what is deployed; the dashboard's rule composer builds the same shape from a failure you clicked on. Full reference, scoring per type, and worked examples: docs/custom-rules.md.

Full tool schemas and configuration: iris-eval.com

Hosted features

Iris runs entirely on your machine today, and everything it does is free and MIT licensed with no limits and no account.

Hosted storage, shared team history and alerting are under consideration, not under construction. There is no pricing, and nothing to buy. If shared history would be useful to you, the waitlist is how we find out whether it's worth building — it commits you to nothing.

Two commitments hold regardless: nothing that is free today will move behind a paywall, and no compliance certification will be claimed before it is held.

Examples

• Claude Desktop setup — MCP config for stdio and HTTP modes
• TypeScript — MCP SDK client — connect and invoke tools
• HTTP transport (TS + Python) — full client code for REST-style integration
• LangChain instrumentation (Python, conceptual) — scaffold showing the shape; needs your agent code to be runnable
• CrewAI instrumentation (Python, conceptual) — scaffold; same caveat

Community

• GitHub Issues — Bug reports and feature requests
• GitHub Discussions — Questions and ideas
• Contributing Guide — How to contribute
• HTTP Ingest — Deterministic trace capture via POST /api/v1/traces
• Roadmap — What's coming next
• Versioning policy — What each version number promises, and what has to be true before 1.0

# Production deployment
iris-mcp --transport http --port 3000 --api-key "$(openssl rand -hex 32)" --dashboard

npx @iris-eval/mcp-server --self-test

npx --yes @iris-eval/mcp-server@latest

npm install -g @iris-eval/mcp-server@latest

npm rebuild better-sqlite3
# for a global install:
npm rebuild -g better-sqlite3

npx @iris-eval/mcp-server --version

# If using npx (clears cache and fetches latest)
npx --yes @iris-eval/mcp-server@latest

# If installed globally
npm update -g @iris-eval/mcp-server

node --version  # Must be v20.x or v22.x+

# Correct
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server

# Wrong (causes /c to be parsed as a path)
claude mcp add --transport stdio iris-eval -- cmd /c "npx @iris-eval/mcp-server"

If Iris is useful to you, consider starring the repo — it helps others find it.

MIT Licensed.