AI evaluation & benchmarks MCP servers

A server here evaluates, tests or benchmarks AI models or agents, or provides benchmark tasks for them.

Not here: testing ordinary software.

752servers
439 remote313 localTagged by Odel's AI, which can make mistakes

Best fit

400 made for this, ranked by starsRemoteLocal
  1. jevnav @dtduc-git/jevnavReplayable browser automation: Jev picks the element, a gate blocks risky actions, every run traces.6% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.2 GitHub stars.
  2. LLM Preflight @feronovak/llm-preflightLocal, cross-provider preflight checks for LLM integration changes.5% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.2 GitHub stars.
  3. Dsh Verify @263311487-ux/dsh-verifyAgent-built web app quality gate. Real browser is the judge. PASS/FAIL with receipts.4% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.2 GitHub stars.
  4. ChimeraForge @sahil170595/chimeraforgeLocal-first LLM deployment planner: GPU/VRAM sizing, cost and latency, with provenance3% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.2 GitHub stars.
  5. Memtrust @rudrendupaul/memtrustIndependent, reproducible benchmark harness for agent-memory backends.3% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.2 GitHub stars.
  6. Hotato @attenlabs/hotatoOpen-source, self-hosted conversation QA for voice agents. MIT.3% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.2 GitHub stars.
  7. Arkheiaai MCP @arkheiaai/mcp-serverDetect fabrication and hallucination in any LLM output. 30+ models supported.15% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.2 GitHub stars.
  8. Testers.ai CARBON Test Harness Lite @jarbon/carbon-liteLocal QA workflows, test evidence, screenshot maps and confidence reports for AI coding agents.Not assessed yet.Runs on your machine, installed from MCPB.1 GitHub stars.
  9. Jev Decision Gate by BPJ @f-tiger/jev-decision-gateLocal Issue triage with optional Jev judgments and calibrated acceptance gates.Not assessed yet.Runs on your machine, installed from MCPB.1 GitHub stars.
  10. Harvey Verify @meltingpixelsai/harvey-verifyPost-transaction outcome verification. LLM-as-judge for agent service quality.Not assessed yet.Runs on your machine, installed from npm.1 GitHub stars.
  11. Groundwork @nickjlamb/groundworkDeployment-readiness checks for document-QA and extraction AI: grounding, extraction, CI gate.Not assessed yet.Runs on your machine, installed from npm.1 GitHub stars.
  12. LLM Redteam @4hmetuyar/llm-redteamActive jailbreak/extraction/obfuscation red-teaming for live LLM endpoints16% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  13. Ticket Demo @markusvankempen/mcp-ticket-demoMCP server demo — tickets, schema discovery, auth, and PII gating. stdio + HTTP.4% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  14. ios-agent @emazaheri/ios-agentDrive a real iPhone or an iOS Simulator with an AI agent, through XCUIAutomation.7% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.1 GitHub stars.
  15. ProfitPlay @jarvismaximum-hue/profitplay-mcpMCP server for the live ProfitPlay BTC five-minute AI agent sandbox9% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from Docker.1 GitHub stars.
  16. Sentinel Scan @ventrova/sentinel-scan-cliMCP security scanner: detect tool poisoning, prompt injection & rug-pulls - 10 OWASP heuristics.7% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  17. RuleDrift @ri7in/rule-driftTest whether AI agents retain critical instructions as conversations grow.5% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  18. Evolveguard @rudrendupaul/evolveguardWraps the evolveguard CLI as a single generic MCP tool for agent-skill file safety checks.3% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.1 GitHub stars.
  19. EdgeGate @frozo-ai/edgegate-mcpRegression-test AI models on real Snapdragon and Jetson hardware, with signed evidence for CI.5% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  20. GAUNTLEX @sanjoy1234/gauntlexConcurrent Builder + Breaker agents generate code and adversarial tests at once, gated in CI.7% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.1 GitHub stars.
  21. QueryPilot @nickklos10/querypilotSafe SQL gateway for AI agents: SELECT-only validation, access policies, masking, audit trail, evals4% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.1 GitHub stars.
  22. Rotifer Protocol MCP @rotifer-protocol/mcp-serverSearch, compare, and install self-evolving AI Agent Genes ranked by Arena fitness7% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  23. Holomime @productstein/holomimeBehavioral therapy for AI agents — self-diagnosis, alignment, and training via MCP7% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  24. Opengate @nickjlamb/opengate-mcpCheck whether an AI answer is grounded in its context — deterministic, no LLM judge.5% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.