overfitting-audit-mcp

An MCP server that answers "is this edge real, or a testing-hundreds-of-variants artifact?" — implementing the Probability of Backtest Overfitting (CSCV method), Deflated Sharpe Ratio, Minimum Backtest Length, and purged/embargoed cross-validation splits.

The problem this solves

Testing enough parameter combinations against the same historical data will eventually produce a great-looking backtest by chance alone. Standard backtest metrics (Sharpe, win rate, profit factor) don't distinguish a genuine edge from the best-looking result out of hundreds of near-identical variants. This audits for that specific failure mode directly, rather than trusting a single strong-looking curve.

Tools

probability_of_backtest_overfitting

Combinatorially Symmetric Cross-Validation (CSCV) method — estimates the probability that a strategy's in-sample performance rank won't hold out-of-sample.

deflated_sharpe_ratio

Adjusts a Sharpe ratio for the number of trials run and the non-normality of returns, so it can't be inflated just by testing more variants.

minimum_backtest_length

The minimum number of independent trials/observations needed before a given Sharpe ratio is statistically meaningful at all.

purged_cv_split

Generates purged and embargoed cross-validation splits for time-series backtests, preventing the lookahead leakage that ordinary k-fold CV introduces on financial data.

Use it

Hosted (recommended): MCPize — free tier, paid Pro tier for higher limits.

Self-host:

npm install
node server.js

Part of the AlgoForge suite

Prop-firm and quant-validation tools for algo traders: prop-rules-mcp, trade-journal-mcp, payout-calc-mcp, econ-calendar-mcp, montecarlo-validator-mcp, walkforward-validator-mcp, pinescript-audit-mcp, backtest-cost-sensitivity-mcp, pinescript-mcp.

License

MIT