digline

Regression testing for LLM applications — with the baseline in your repository, not on someone's server.

Your prompt worked on Tuesday. On Thursday it works a little less — not enough to break, enough for a user to notice in two weeks. No ordinary test catches it, because there is no correct output to compare against, only a better or a worse one.

digline gives you an approved reference — the baseline — and on every change tells you whether you are below it: which case, which check, by how much. The baseline is a JSON file in your repository, so it goes through code review and it rolls back with git. No server, no account, no network call you have not configured yourself.

$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).

Why digline

Most evaluation tools tell you whether an output is below a threshold. digline also tells you whether it is worse than it was — the drift from 0.91 to 0.78 that trips no threshold and is the first thing a user feels.

The suite is Python, not YAML: a judge is an object, a target is a function, and what may leave a perimeter is declared in code — none of which a configuration file expresses without reinventing a language. Where a suite is plain data it can be TOML instead, and the two forms build the same objects — what TOML cannot express, it refuses by name rather than half-supporting. Built for teams shipping LLM features for someone else, who have to show a customer what was tested, when, under which commit, and who approved it.

Wondering how digline differs from promptfoo, DeepEval, or observability platforms? See How digline compares.

Quickstart

With uv (recommended):

uv init && uv add digline

or with pip in an existing environment: pip install digline.

Requires Python 3.12+, which uv fetches for you if you do not have it. On the pip path an older interpreter says ERROR: No matching distribution found with from versions: none, which does not say why — that is what it means.

If you use Claude Code, digline also installs as a plugin. What it ships first is the judgement layer. The operating-digline skill carries the same rules as AGENTS.md, and loads when the agent is about to run a suite or read a comparison. Beside it is the MCP server, which measures, reads and explains, and cannot promote a baseline. Two lines plus one dependency: the server imports your suite, so it runs from your project's .venv.

uv add --dev digline-mcp
claude plugin marketplace add digline/digline
claude plugin install digline@digline --scope project

The agent's own shell commands are not the server. A bare digline typed there resolves through the PATH of the shell Claude Code started, which comes from your login profile and from whatever environment was active when you launched it, and it can be another project's install at another version. uv run digline resolves through your project's environment.

What it installs, and what its hook does and does not guarantee: plugins/digline/README.md.

In another agent, the skill alone installs through skills.sh: npx skills add digline/digline --skill operating-digline. That is the skill without the MCP server or the hook, taken from main rather than a release.

suite.py — complete and runnable, no API key:

"""suite.py — complete and runnable: no API key, nothing else to install."""

from digline.core import Contains, CostBudget, JudgeReply, LlmRubric
from digline.run import Case, Response, Suite

ANSWERS = {
    "where-is-my-order": "Order 4821 ships Thursday. — Northwind Support",
    "how-do-i-return": "Any item, within 30 days, unused. — Northwind Support",
}


def judge(prompt: str) -> JudgeReply:
    """Your judge. digline composes `prompt` from the rubric, the question and
    the answer; it wants a score in [0, 1] and a reason back."""
    signed = "Northwind Support" in prompt
    concise = len(prompt.split()) <= 60
    return JudgeReply(
        score=0.4 + 0.3 * signed + 0.3 * concise,
        reason=f"signed={signed}, concise={concise}",
    )


def target(case: Case) -> Response:
    """Your application, called once per case. Canned here so this runs as is."""
    text = ANSWERS[case.id]
    return Response(output=text, cost_usd=0.004 + 0.001 * len(text) / 100)


suite = Suite(
    tenant="northwind",
    environment="staging",
    name="support",
    assertions=[
        Contains(needle="Northwind Support"),
        LlmRubric(
            rubric="Does the reply answer the question in at most three sentences?",
            judge=judge,
            threshold=0.7,
            tolerance=0.05,
        ),
        CostBudget(max_usd=0.02, tolerance=0.05),
    ],
    cases=[Case(id="where-is-my-order"), Case(id="how-do-i-return")],
)

tenant="northwind" is ours, like every tenant in this repository's examples, and it has to become yours before the first run. A tenant is one customer: one perimeter, and the directory your store is keyed on, .digline/<tenant>/. Name it after the customer the suite is for. A suite copied to the next customer with this line unchanged files that customer's runs in the first one's directory, compares them against the first one's reference and promotes over it, each with exit 0.

When your judge is a real model, add a provider plugin: uv add digline-anthropic (or pip install digline-anthropic), likewise digline-openai and digline-bedrock.

$ digline run --suite suite.py
2026-08-26T15-44-09-282929-00-00-e7421ec503ccefe8

$ digline promote --suite suite.py --run latest --replacing none
tenant 'northwind' · support baseline set to 2026-08-26T15-44-09-282929-00-00-e7421ec503ccefe8

Now make it worse — delete — Northwind Support from the second answer — and ask again:

$ digline run --suite suite.py
2026-08-26T15-44-09-492722-00-00-e7421ec503ccefe8

$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).

$ echo $?
1

The exit code is the answer: 0 fine, 1 got worse, 2 could not be judged. Everything lands in .digline/<tenant>/ — baselines/ committed, runs/ git-ignored through a .gitignore digline writes for you.

What it checks

Per case — pure functions (inputs) -> Verdict, no I/O, callable on their own:

A suite pointed at an OpenAI-compatible endpoint, a gateway or a self-hosted model declares what it actually charges — [target.pricing] in TOML, override() in Python — so a CostBudget reads your prices rather than the plugin's list.

Per run — one verdict on the whole suite, the kind that goes in a contract:

Every assertion carries a threshold that can fail — there is no default that passes vacuously, and Contains("") is a ValueError when the suite loads rather than a green run — and a tolerance below which a difference from the baseline is noise. Where a number is really "k out of n", write it as one: min_agreement="2/3", and a float no k/n can produce is refused at construction.

One card each — parameters, typical values, what to watch out for — in docs/metrics.md. Custom assertion? Subclass AssertionBase, or RunAssertionBase for an aggregate: docs/api.md.

How it thinks

• The judge is yours. The core never calls a model API: you inject a function or a provider plugin's judge, and in your tests a deterministic one.
• Three states, not two — pass, fail, error. An error is neither green nor a regression: it means could not judge, and a run containing one cannot become the baseline.
• Two kinds of noise, two answers. Suite.samples asks the target more than once — the same input answered differently. Repeated grades the same output more than once — the judge changing its mind. min_agreement becomes mandatory as soon as you sample.
• A tolerance is declared; a noise floor is measured. A sampled run records the interval its own samples spanned, and a drop that stays inside the baseline's interval is reported as unchanged rather than as a regression — a tool that cries wolf on its own measurement error teaches people to promote past it. It never rescues a flip, and it never invents an interval it does not have.
• An alias is a pointer, and pointers roll. compare names the model the provider said answered when it changes, digline log reads that down every stored run, and a Case(canary=True) watches behaviour where the provider says nothing: if it moves at all, the headline says the model under the alias likely changed and the run exits 1.
• Set the threshold where the system measurably is, not where you want it: the gate protects against getting worse, and raising the bar is a visible change in a pull request.
• Promote the median of several runs, not the first green one — digline view is the table you pick it from. Cases diagnose, aggregates gate.

Worked through with real numbers in docs/guide.md; the reasoning behind every fixed decision is in docs/adr/.

Commands

The same comparison reaches an agent through digline-mcp — eight tools that read and measure, and no promote to call — and Claude Code through the plugin, which ships that server with the judgement layer beside it. It reaches a test run through pytest-digline, one row per check, a pull request through digline/digline-action, and CI without a Python toolchain through ghcr.io/digline/digline.

Examples

The projects in examples/ each answer a question somebody actually arrives with. Every one runs with no API key, carries its committed report.html, and is a standalone project: copy the directory anywhere and uv sync works.

• I have a classifier: how do I keep it under control? — labelled cases, an agreement check, Precision and Accuracy as the gate
• I'm writing a prompt and have no application yet — a prompt in a file, and the report showing its diff next to what it moved
• I have a RAG: how do I check it doesn't make things up? — frozen retrieval, Faithfulness, PiiAbsent
• My application is Java: can I use this? — HttpTarget against a service digline cannot import
• My app is LangChain4j: what do I put in my repo? — the walkthrough: one endpoint, three files, the CI gate
• My pipeline is LangChain: what changed when I upgraded it? — the chain called in process, FakeListChatModel in CI, one line to a real model
• My agent calls the right tools, but with the right arguments? — a LangGraph agent judged on its trajectory: ToolsCalled for the order, ToolCalledWith for the arguments, the tools real and the model scripted
• My RAG is LlamaIndex: is it still answering from the right page? — a live query engine, retrieval measured by Faithfulness against the page each case declares
• My team does not write Python: can we still gate a prompt? — a suite.toml and a cases.json, no code in the suite
• My suite is green today: who watches it on Thursday? — the operator loop: a scheduled re-run, draw told from drift, a declared policy that decides who is woken, and a probe proving each cycle that promote is still absent
• My agent uses an MCP server I don't control: how do I know its tool descriptions haven't changed? — a canonical tools/list dump declared as an artifact, and one added clause that stops a tool being called

What digline is not

• Not an observability platform. Dashboards over production traces are a served market. What is designed and not yet built is narrower: evaluating production responses inside your perimeter, and turning a failure into a committed test case.
• Not a red-teaming tool. digline generates no attacks. Once one is found, it becomes a Case, and the suite makes sure it never works again.
• Not YAML. The suite is Python — or, within declared limits, TOML: cases were always data, and now the rules can be too. A judge with rules of its own, a computed request body and a custom assertion stay Python, and the loader names the wall you hit rather than half-supporting it.
• Not a funnel. Two commitments, by design and for good: no hosted service that receives your payloads, and no data collection. The baseline lives in your repo; the runs happen on your machines. If digline ever grows paid features, they will run inside your perimeter too. How digline treats its own attack surface — the published advisories and the delta-pass every minor release gets — is in SECURITY.md.

Status

0.31.0, pre-1.0. The offline cycle — write the suite, run, compare, promote, report, and commit what a person decided — is complete, covered by tests, and used daily on a real project, and since 0.5.0 the suite may be written as data as well as in Python. The API may still change before 1.0; the baseline format is versioned and migrates. The production store, the bridge from production failures back to committed cases, and the reactive side are designed in ADR 0002 and not written yet.

Python 3.12+. One runtime dependency: jsonschema.

Docs

• docs/guide.md — how to reason with digline, in eight chapters and the order the problems arrive: baseline, judge noise, sampling, tolerance, threshold, which run to promote, what to gate on, what to maintain
• docs/metrics.md — a card per assertion and aggregate: when to reach for it, what it produces, what it will do to you if you are not looking
• docs/api.md — what is imported from where, every assertion and its parameters, custom assertions, and the complete example in examples/quickstart/, which a test runs on every build
• docs/declarative.md — the suite as data: the TOML format, key by key, what it deliberately cannot say, and how to move a suite between the two forms without losing its baseline
• docs/diff.md · docs/explain.md · docs/rejudge.md · docs/view.md · docs/migrate.md — each command with a surface of its own
• docs/mcp.md · docs/pytest.md — the two front ends that are not the CLI
• AGENTS.md — how a coding agent should operate digline in your repo
• plugins/digline/ — the same rules as a Claude Code plugin, with the MCP server and a hook that asks a person before promote or register
• docs/adr/ — the architectural decisions, numbered, with the reasoning

License

Apache-2.0.