AI evaluation & benchmarks MCP servers

A server here evaluates, tests or benchmarks AI models or agents, or provides benchmark tasks for them.

Not here: testing ordinary software.

752servers
439 remote313 localTagged by Odel's AI, which can make mistakes

Best fit

400 made for this, ranked by starsRemoteLocal
  1. AWT (AI Watch Tester) @ksgisang/awtAI-powered E2E testing MCP server. Detects and auto-fixes UI bugs via DevQA Loop and Vision AI.6% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.1 GitHub stars.
  2. Trustmodel @karlmehta/trustmodel-mcpScore any AI for trust across 10 dimensions; evaluate, monitor & govern LLMs and agents.5% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  3. Index9 @index9-org/mcpDiscover, shortlist, compare, cost-model, and live-test 300+ AI models from your editor5% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  4. Evals @cyanheads/evals-mcp-serverAuthor verifiable eval records through a draft→review→revise→submit loop with enforced graders.3% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.1 GitHub stars.
  5. Datumline Agent Guard @datumline/agent-guardChecks an AI agent's 'done' against the files and URLs it claims. Missing = FALSE_DONE.3% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.
  6. Unicode Logic Kit @fvossel/unicode-logic-kitParse, check, prove and translate logic formulas: first-order, modal, description, higher-order.Not assessed yet.Runs on your machine, installed from PyPI.
  7. Dunstan @barglabs/dunstanChecks a coding agent's handback against the pull request's GitHub record. No model decides.3% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.
  8. Vibe Gate @mustafacagri/vibe-gate-mcpAdversarial Quality Gate MCP for vibe-coding: IDE AI vs Critic AI, human decides on deadlockNot assessed yet.Runs on your machine, installed from npm.
  9. Calibration Explorer @m-sanchez/calibration-explorerCheck classifier confidence against accuracy, fit temperature scaling and save local reports.Not assessed yet.Runs on your machine, installed from npm.
  10. Technical Answer Validator @christofer566/technical-answer-validatorRubric-based technical answer feedback for AI agents; human review required.5% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.
  11. Pramana @loopg/pramana-mcpRead your recorded AI agent runs, and what changed when you swapped a model or prompt.Not assessed yet.Runs on your machine, installed from PyPI.
  12. honesty-mcp @lkopietz3-byte/honesty-mcpMCP server exposing 11 honesty-SDK kits (grounding, trust, audit, corroboration) as agent tools.Not assessed yet.Runs on your machine, installed from npm.
  13. Groundline @fhs1220888/groundlineVerifiable test-data analysis: deterministic tools, evidence ledger, number-level checks.Not assessed yet.Runs on your machine, installed from PyPI.
  14. LEEVAR reliability battery @forevercrab321-svg/leevarGrade an AI agent's transcripts on 18 reliability tests. Thin evidence is NOT TESTED, not guessed.7% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from MCPB.
  15. PolyAccounts synthetic accounting demo @andrewblount/polyaccounts-demoSynthetic accounting MCP demo with exact balances and ledger evidence. No account or API key.4% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from MCPB.
  16. Ragmetric @mukundakatta/ragmetric-mcpRAG retrieval IR metrics: recall@k, hit@k, MRR, NDCG@k, evaluate_batch.3% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.
  17. Kineti @therawlogs/kinetiKineti: spend cap, undo, and test proof for AI coding agents6% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.
  18. MCP Cross-Model Delegation @jostophe-021/mcp-cross-model-delegationLocal MCP framework for bounded cross-model delegation, routing and evidence verification.4% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from Docker.
  19. B2IGE Verify @b2ige2-a11y/b2ige-verifyIndependent deterministic verification for AI-written software. Local, evidence-backed checks.5% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from MCPB.
  20. IsolatedTester — TinkyVision system @agewellepm/isolated-testerIsolated macOS app tests, MCP images, OCR/ASCII frames, accessibility and evidence reports.12% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from MCPB.
  21. Ami Survey @speedofred/ami-surveyMeasures the cost, tokens and duration of a completed agent workflow from its session log.4% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.
  22. Rechtssysteem.ai @rechtssysteem/rechtssysteem-mcpLekkage-vrije voorspelling van Nederlandse rechtszaken, getraind op 609.715 uitspraken9% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from a package registry.
  23. Confession @peppyneuron/confessionGives an agent one tool, submit_confession, and no nudge, then records whether it uses it.11% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from npm.
  24. Benchmark Hygiene @sudo-ai-git/mcp-benchmark-hygieneDetect pytest config-leakage that corrupts agent-benchmark grading. Deterministic, no-LLM.4% chance this is malicious. Assessed by AI, which can make mistakes.Runs on your machine, installed from PyPI.