[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_124":125},"routes/modules.$creator.$moduleId",{"_14":15,"_16":17,"_18":19,"_20":21},"routes/modules.$creator.$moduleId.endpoints",{"_11":12,"_13":-5},"error","Module not found","tools","module",{"_20":22,"_23":21,"_24":25,"_26":27,"_28":29,"_30":31,"_32":33,"_34":25,"_35":36,"_37":-5,"_38":-5,"_39":-5,"_40":41,"_42":43,"_44":45,"_46":47,"_48":49,"_50":51,"_52":53,"_54":-5,"_55":47,"_56":57,"_58":59,"_60":-5,"_61":-5,"_62":-5,"_63":-5,"_64":65,"_66":67,"_68":69,"_70":71,"_72":73,"_74":-5,"_75":-5,"_76":-5,"_77":47,"_78":-5,"_79":80,"_81":82,"_83":84,"_85":86,"_87":88,"_89":90},"apiUrl","https://odel.app/api","creator","rudrendupaul","moduleId","memtrust","@rudrendupaul/memtrust","displayName","tagline","Independent, reproducible benchmark harness for agent-memory backends.","iconKey","https://avatars.githubusercontent.com/u/38769913?v=4","category","local","status","published","visibility","public","metaDescription","deploymentType","external","workerName","bindingName","mcpConfig","pricingModel","free","creatorUserId","system:external-import","createdAt",1786422398000,"updatedAt",1788328931000,"sourceRegistry","official-mcp-registry","sourceUrl","https://registry.modelcontextprotocol.io/v0/servers?search=io.github.RudrenduPaul%2Fmemtrust","sourceVerified",true,"sourceMissingSince","importedAt","upstreamStars",1,"upstreamUpdatedAt",1787672704000,"detectedTransport","detectedAuthType","probeStatus","lastProbedAt","githubRepoUrl","https://github.com/RudrenduPaul/memtrust","githubRepoFullName","RudrenduPaul/memtrust","releaseId","ext-rudrendupaul-memtrust","version","0.3.6","descriptionMarkdown","# memtrust\n\n\n\n\n\n[![CI](https://github.com/RudrenduPaul/memtrust/actions/workflows/ci.yml/badge.svg)](https://github.com/RudrenduPaul/memtrust/actions/workflows/ci.yml)\n[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/RudrenduPaul/memtrust/blob/main/LICENSE)\n[![PyPI Version](https://img.shields.io/pypi/v/memtrust-cli)](https://pypi.org/project/memtrust-cli/)\n[![npm Version](https://img.shields.io/npm/v/memtrust-cli)](https://www.npmjs.com/package/memtrust-cli)\n[![PyPI](https://img.shields.io/badge/pypi-memtrust--cli-blue.svg)](https://pypi.org/project/memtrust-cli/)\n\n $\"memtrust$ \n\nAgent memory backends each publish their own benchmark numbers, on different tests, measured\ndifferent ways. memtrust runs the same evals against all four and publishes the raw logs. Run\nagainst the vendors, not by them.\n\n![Terminal recording of installing memtrust-cli with pip into a clean virtualenv, then running memtrust run against all four tracked backends with no credentials configured -- every backend reports SKIPPED and a JSON report is still written.](https://raw.githubusercontent.com/RudrenduPaul/memtrust/main/docs/demo.gif)\n\n```bash\npip install memtrust-cli\nmemtrust run --backends mempalace,mem0,zep,openviking --eval all\n```\n\n(For contributing to this repo instead of just running it, see [Development](#development) --\n`pip install -e \".[dev]\"` from a clone.)\n\n**Contents:** [Why this exists](#why-this-exists) · [What it does](#what-it-does) ·\n[Commands](#commands) · [How this differs](#how-this-differs-from-trusting-a-vendors-own-numbers) ·\n[Contradiction detection](#the-eval-that-actually-matters-contradiction-detection) ·\n[Compression fidelity](#the-eval-built-for-the-other-headline-overclaim-compression-fidelity) ·\n[Temporal-KG boundary](#the-eval-built-from-mempalaces-own-bug-temporal-kg-boundary-detection) ·\n[The landscape](#the-landscape-verified-not-benchmarked) · [Benchmarks](#benchmarks) ·\n[GitHub Actions usage](#github-actions-usage) · [Self-host](#self-host) · [Install](#install) ·\n[Hosted layer](#what-a-hosted-trust-layer-would-add) · [Backend coverage](#backend-coverage) ·\n[Development](#development) · [FAQ](#faq) · [License](#license) · [Success stories](#success-stories)\n\n## Why this exists\n\nIf you've compared agent-memory backends recently, you've probably noticed each one leads with a\ndifferent accuracy number, on a different benchmark, measured a different way. MemPalace's own\ncommunity already flagged the problem in public. Issue [#27](https://github.com/MemPalace/mempalace/issues/27)\non the MemPalace repository, opened April 7, 2026 and still open, documents that a headline 100%\nLongMemEval figure, measured with Haiku reranking, wasn't reproducible from the repository's own\nbenchmark scripts and was pulled from the README as unverifiable. A separate 96.6% figure people\ncite everywhere turns out to be mostly ChromaDB's default embeddings doing the work in raw mode,\nnot MemPalace's own architecture. A \"lossless\" compression claim (the \"AAAK\" mode) drops the same\nLongMemEval score from 96.6% to 84.2% in practice, a 12.4 percentage point gap. Two internal pull\nrequests attempting to fix the reporting problem, #433 and #729, were both closed without merging\non April 12, 2026 -- #729 within seven minutes of being opened. As of this writing, the issue has\n232 thumbs-up reactions and 39 comments.\n\nNone of that means MemPalace, or any other backend, doesn't work. It means nobody outside the\nvendor had run the same test, the same way, against every option, and published the raw logs.\n\nmemtrust does that. It runs LongMemEval, LoCoMo, and a growing set of evals built specifically for\nthis project -- 17 of them as of this writing, all registered in the CLI's `--eval` flag. The two\nthat matter most for understanding what this project is actually for:\ncontradiction detection, because neither LongMemEval nor LoCoMo tests the question that actually\nmatters once a memory system sits underneath a production agent -- what happens when a new fact\ncontradicts an old one? Does the backend flag the conflict? Silently overwrite the old fact with no\naudit trail? Serve whichever version it happens to retrieve first? None of the four backends this\nproject tracks publish a number for that. And compression/round-trip fidelity, built to directly\ntest claims like the \"lossless\" one above: it stores content, retrieves it, and scores literal\nreconstruction fidelity rather than semantic accuracy, per operating mode a backend exposes (see\n`MemoryBackendAdapter.supported_modes`) -- the mechanism that would let a contributor with live\nMemPalace credentials actually reproduce the 12.4-point compressed-mode accuracy drop\nmempalace/mempalace#27 documents, instead of just citing it. **Neither has been run against a live\nMemPalace instance as of this writing** -- both have, however, been run against a live\nself-hosted `mem0ai` install; see \"Benchmarks\" below. The other evals -- ranking quality,\ncrash recovery, extraction quality, embedding drift, scale/volume stress, lock contention, stats\naccuracy, orphan cleanup, result consistency, migration rollback, filter injection, resource-sync\nsafety, and temporal-KG boundary detection -- each grew out of a specific real bug report against\none of the four tracked backends; see \"Success stories\" below for the full list.\n\n**Where this stands right now, in one place:** [live benchmark results](#benchmarks) for one\nbackend (`mem0_direct`, self-hosted `mem0ai`), including a real bug this project's own attempt to\nget those numbers surfaced in mem0's default configuration; and [197 real GitHub issues and PRs](#success-stories)\nfiled against MemPalace, Mem0, Zep/Graphiti, and OpenViking independently root-caused against this\ncodebase -- 55 (28%) PASS, 16 (8%) PARTIAL, 42 (21%) a genuine capability gap, 84 (43%) not\napplicable, every verdict re-verified by a reviewer independent of whoever built the fix.\n\n## What it does\n\nEvery command below was actually run against this repo, with zero vendor API keys configured, to\nproduce the output shown. Nothing here is simulated.\n\n```\n$ memtrust run --backends mempalace,mem0,zep,openviking --eval all\nmemtrust 0.3.4 -- run_id=mt_2026-08-04T061759Z\nBackends: mempalace, mem0, zep, openviking Evals: longmemeval, locomo, contradiction,\nresource_sync_safety, compression, ranking_quality, scale_stress, embedding_drift, crash_recovery,\nextraction_quality, migration_rollback, filter_injection, lock_contention, stats_accuracy,\norphan_cleanup, result_consistency, temporal_kg_boundary\n\nmempalace: SKIPPED (not configured) -- mempalace is not configured: environment variable\nMEMPALACE_STORAGE_PATH is not set. Skipping this backend. See docs/methodology.md for setup\ninstructions.\nmem0: SKIPPED (not configured) -- mem0 is not configured: environment variable MEM0_API_KEY is not\nset. Skipping this backend. See docs/methodology.md for setup instructions.\nzep: SKIPPED (not configured) -- zep is not configured: environment variable ZEP_API_KEY is not set.\nSkipping this backend. See docs/methodology.md for setup instructions.\nopenviking: SKIPPED (not configured) -- openviking is not configured: environment variable\nOPENVIKING_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup\ninstructions.\n\nCost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)\n\nFull report: memtrust-report-2026-08-03.json\n```\n\nThat's the real, reproducible behavior of a fresh clone with no credentials: every backend reports\nSKIPPED, the command exits cleanly, and a valid JSON report is still written. `memtrust --version`\nnow correctly prints the installed version, matching `pip show memtrust-cli`. Earlier releases printed\n`0.0.0+unknown` even when properly installed, because `src/memtrust/__init__.py` read\n`importlib.metadata.version(\"memtrust\")` while the installed distribution is actually named\n`memtrust-cli` -- kept in the FAQ below for the record rather than deleted, since silently erasing\na bug the moment it's fixed is exactly the kind of curation this project exists to push back on in\nother people's benchmarks. Set the relevant environment variable for any backend you want to\nactually test (`MEM0_API_KEY`, `ZEP_API_KEY`, `OPENVIKING_API_KEY`, `MEMPALACE_STORAGE_PATH`) and\nthat backend runs for real against its live API instead of being skipped.\n\nThe eval logic itself is proven offline, against the bundled synthetic fixtures and, for several\nadapters, the real installed vendor packages with only the network boundary mocked, by the test\nsuite:\n\n```\n$ pytest --cov=memtrust --cov-report=term-missing\n... (33 module rows total; the 11 most relevant to this README are shown below)\nName Stmts Miss Cover\n-------------------------------------------------------------------------------------\nsrc/memtrust/adapters/base.py 290 1 99%\nsrc/memtrust/adapters/mempalace_adapter.py 265 15 94%\nsrc/memtrust/adapters/mem0_adapter.py 140 12 91%\nsrc/memtrust/adapters/mem0_direct_adapter.py 281 34 88%\nsrc/memtrust/adapters/openviking_adapter.py 178 18 90%\nsrc/memtrust/adapters/zep_graphiti_adapter.py 63 3 95%\nsrc/memtrust/adapters/zep_graphiti_selfhosted_adapter.py 165 24 85%\nsrc/memtrust/evals/contradiction.py 127 2 98%\nsrc/memtrust/evals/compression.py 86 1 99%\nsrc/memtrust/evals/temporal_kg_boundary.py 90 3 97%\nsrc/memtrust/receipt.py 118 10 92%\n-------------------------------------------------------------------------------------\nTOTAL 4167 265 94%\n\n590 passed, 8 skipped in 5.92s\n```\n\nThis is an excerpt, not the full table -- the weakest-covered module in the repo,\n`evals/mempalace_metadata_scale.py` (70%), isn't one of the 11 shown above; run the command\nyourself for the complete per-module breakdown.\n\n590 passing tests across 33 source modules, 94% overall statement coverage, 98% on the\ncontradiction-detection eval, 99% on compression/round-trip fidelity, 97% on the temporal-KG\nboundary eval, 85-99% across the adapter layer. The 8 skips are live-`mempalace`-package tests that\nonly run with the optional `mempalace-direct` extra installed (`pip install -e\n'.[dev,mempalace-direct]'`). Every test mocks its HTTP or wire\nlayer, or uses an in-memory fake backend -- none of them touch a real network, though a meaningful\nshare of the adapter tests now import and exercise *real installed vendor classes* directly\n(`mem0ai==2.0.12`'s embedder and vector-store modules, and -- gated behind the optional\n`mempalace-direct` extra -- the real `mempalace.mcp_server` functions), mocking only the outermost\nnetwork or wire-client boundary rather than the whole library. `graphiti-core` is not installed in\nthis environment, so its self-hosted adapter's tests still run against a hand-written Protocol\ndouble built to match the real package's confirmed method signatures, not the real classes -- see\n`docs/methodology.md`'s adapter confidence table for exactly which claim rests on which kind of\nverification.\n\n![Terminal recording of memtrust run against all four tracked backends with zero credentials configured, showing every backend report SKIPPED and a valid JSON report still get written.](https://raw.githubusercontent.com/RudrenduPaul/memtrust/main/docs/assets/dev-to-demos/demo-1-zero-credentials.gif)\n\n## Commands\n\n```\n$ memtrust --help\nUsage: memtrust [OPTIONS] COMMAND [ARGS]...\n\n memtrust: an independent, reproducible benchmark harness for agent-memory\n backends.\n\nOptions:\n --version Show the version and exit.\n --help Show this message and exit.\n\nCommands:\n keygen Generate a new Ed25519 keypair for signing `memtrust run`...\n report Read a prior `memtrust run` JSON report and print a formatted...\n run Run the eval suite against the requested backends.\n verify Verify a signed receipt produced by `memtrust run --sign`.\n```\n\n| Command | Flags | What it does |\n|---|---|---|\n| `memtrust run` | `--backends TEXT` comma-separated list or `all` (default `all`) · `--eval TEXT` comma-separated from `longmemeval,locomo,contradiction,resource_sync_safety,compression,ranking_quality,scale_stress,embedding_drift,crash_recovery,extraction_quality,migration_rollback,filter_injection,lock_contention,stats_accuracy,orphan_cleanup,result_consistency,temporal_kg_boundary`, or `all` (default `all`) · `--output FILE` (defaults to `./memtrust-report-.json`) · `--locomo-dataset-path FILE` points the LoCoMo eval at a real, downloaded `locomo10.json` instead of the bundled synthetic fixture (memtrust does not bundle or auto-fetch the real dataset) · `--locomo-exclude-question-ids-file FILE` excludes known-bad-ground-truth LoCoMo question IDs from scoring · `--scale-stress-n-records INTEGER` (default `500`) sets how many synthetic records the scale-stress eval stores and re-queries · `--sign FILE` writes a signed `.receipt.json` alongside the report, proving it was produced by the holder of the given Ed25519 private key | Runs the eval suite against the requested backends. A backend without its credential env var set prints `SKIPPED` and the run continues -- this command never crashes on missing credentials. `temporal_kg_boundary` only applies to the `mempalace` backend (the only adapter that wires `kg_add`/`kg_invalidate`/`kg_query`); requesting it against any other backend reports `not_applicable`, not an error. |\n| `memtrust report REPORT_PATH` | positional path to a prior JSON report · `--json` prints the parsed report as JSON instead of a formatted summary | Reads a report written by `memtrust run` and prints a formatted summary. |\n| `memtrust keygen` | `--private-key-out FILE` (default `memtrust-key.pem`) · `--public-key-out FILE` (default `memtrust-key.pub`) · `--force` overwrites existing output files | Generates a new Ed25519 keypair for signing reports with `run --sign`. |\n| `memtrust verify RECEIPT_PATH` | `--public-key FILE` (or the `MEMTRUST_RECEIPT_PUBLIC_KEY` env var) · `--json` prints the result as JSON | Verifies a signed receipt produced by `memtrust run --sign`; a tampered or mismatched receipt fails verification. |\n| `memtrust --version` | -- | Prints the installed version. |\n\nEvery line above came straight from running `memtrust --help`, `memtrust run --help`,\n`memtrust report --help`, `memtrust keygen --help`, and `memtrust verify --help` against this\nrepo. Nothing here is invented.\n\n![Terminal recording walking the full memtrust CLI surface: memtrust --help, then each subcommand's own --help output for run, report, keygen, and verify.](https://raw.githubusercontent.com/RudrenduPaul/memtrust/main/docs/assets/dev-to-demos/demo-3-cli-surface.gif)\n\n![Terminal recording of memtrust keygen generating an Ed25519 keypair, memtrust run --sign producing a signed receipt from a real run, and memtrust verify confirming the receipt's signature is valid.](https://raw.githubusercontent.com/RudrenduPaul/memtrust/main/docs/usage.gif)\n\n## MCP Server\n\nmemtrust ships a [Model Context Protocol](https://modelcontextprotocol.io) server so an AI agent\n(Claude, Cursor, or any MCP-compatible client) can run a memory-backend benchmark directly, without\na human invoking the CLI by hand.\n\nInstall the extra:\n\n```bash\npip install \"memtrust-cli[mcp]\"\n```\n\nAdd it to your MCP client's config (for Claude Desktop, `claude_desktop_config.json`):\n\n```json\n{\n \"mcpServers\": {\n \"memtrust\": {\n \"command\": \"uvx\",\n \"args\": [\"--from\", \"memtrust-cli\", \"memtrust-mcp\"]\n }\n }\n}\n```\n\nThe server exposes one tool, `run`, that shells out to `memtrust run` with the given arguments\n(memtrust has no `--json` flag, so the wrapper writes to a private temp `--output` file and reads\nit back) and returns the parsed JSON report:\n\n```\nrun([\"--backends\", \"mempalace\", \"--eval\", \"stats_accuracy\"])\n```\n\nTransport is stdio, so there is nothing to host: the MCP client spawns the server as a local\nsubprocess. Source: [`src/memtrust/mcp_server.py`](https://github.com/RudrenduPaul/memtrust/blob/main/src/memtrust/mcp_server.py).\n\n## How this differs from trusting a vendor's own numbers\n\nEvery backend memtrust tracks publishes its own benchmark numbers. None of them publish the same\nbenchmark, scored the same way, with the same held-out discipline. memtrust doesn't ask you to\ntrust it instead: it asks you to read the raw logs. Every run's methodology, prompt templates,\ndataset versions, and scoring rubric are published in `docs/methodology.md`, versioned alongside\nthe code that produced them. If the methodology has a flaw, it's a flaw you can point to in a\nspecific file and line, not something buried in a vendor's internal eval pipeline.\n\nGeneral-purpose LLM eval frameworks (promptfoo, DeepEval, RAGAS, and similar tools) are mature and\nwidely used, but none of them ship a memory-backend adapter abstraction or a contradiction-\ndetection eval out of the box -- they're built for RAG quality, red-teaming, and general prompt\nevaluation, not for comparing how different memory systems handle a fact that changes over time.\nmemtrust is narrower and more specific on purpose.\n\n## The landscape (verified, not benchmarked)\n\nReal, publicly checkable numbers as of this writing (`gh api repos//`), not\nmemtrust-run scores -- accuracy and contradiction-handling comparisons stay in the \"Benchmarks\"\nsection below until a live run actually produces them:\n\n| Backend | GitHub stars | Self-reported description |\n|---|---|---|\n| [MemPalace](https://github.com/MemPalace/mempalace) | 58,032 | \"The best-benchmarked open-source AI memory system. And it's free.\" |\n| [Mem0](https://github.com/mem0ai/mem0) | 62,450 | \"Universal memory layer for AI Agents\" |\n| [Zep / Graphiti](https://github.com/getzep/graphiti) | 29,526 | \"Build Real-Time Knowledge Graphs for AI Agents\" |\n| [OpenViking](https://github.com/volcengine/OpenViking) | 27,859 | \"Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.\" |\n\nNone of these numbers say anything about which backend handles a contradicted fact correctly --\nthat's the whole reason the harness exists. Star count measures adoption, not correctness.\n\n## The eval that actually matters: contradiction detection\n\nLongMemEval and LoCoMo both measure recall: can the backend remember a fact you told it earlier.\nThat's necessary but not sufficient. The harder question is what a backend does when two facts\nconflict: you tell it your meeting is at 2pm, then later say it moved to 3pm. Does it flag the\nchange? Overwrite silently? Serve whichever one it retrieves first? `memtrust`'s classifier stores\na fact, stores a contradicting fact, queries for it, then checks the actual retrieved content for\nboth values, rather than trusting whatever conflict signal the adapter itself reports. See\n`src/memtrust/evals/contradiction.py` and the scoring-logic section of `docs/methodology.md` for\nexactly how that classification works.\n\n## The eval built for the other headline overclaim: compression fidelity\n\nmempalace/mempalace#27 documents two separate overclaims, not one: the LongMemEval score gap\ndescribed above, and a \"lossless\" compression claim that measured 12.4 percentage points lower in\npractice under a compressed operating mode. memtrust could not previously reproduce that second\nnumber at all -- there was no way to tell an adapter \"run this under mode X vs mode Y\" through the\nshared interface. `MemoryBackendAdapter.store()`/`query()` now accept an optional `mode: str |\nNone` parameter, and `MemoryBackendAdapter.supported_modes` lets an adapter declare which mode\nstrings it actually understands (`MemPalaceAdapter.supported_modes` is `(\"raw\", \"AAAK\")`, the two\nnames mempalace/mempalace#27 itself uses -- see `src/memtrust/adapters/mempalace_adapter.py` for\nthe exact provenance and confidence caveat on those names). Adapters with no mode variants accept\nand ignore the parameter, so this is a purely additive, backward-compatible interface change.\n\n`src/memtrust/evals/compression.py` runs the same store-then-retrieve round trip once per mode a\nbackend reports, and scores each round trip with a direct, deterministic character-level\nsimilarity ratio (`fidelity_ratio()`, via `difflib.SequenceMatcher` -- not an LLM judge, since a\n\"lossless\" claim is a literal-reconstruction claim, not a semantic one). This is what would let a\ncontributor with live MemPalace credentials point `memtrust run --eval compression` at it and\nreproduce a \"raw vs AAAK\" fidelity gap directly. **As of this writing this eval has not been run\nagainst a live MemPalace instance** -- it has been run against a live self-hosted `mem0ai` install\n(mean fidelity 31.6%, see \"Benchmarks\" below); see `docs/methodology.md` for the same\nlive-credentials caveat that applies to every other eval and backend not yet measured live.\n\n## The eval built from MemPalace's own bug: temporal-KG boundary detection\n\nMemPalace/mempalace#1913 (fixed by merged PR#1914, contributor ggettert) described a real,\nconcrete bug: `_temporal_filter_sql`'s `as_of` point-in-time query used a closed interval on both\nends, so a fact whose `valid_to` equaled the query's exact `as_of` instant still matched. Hand-roll\na fact change as `kg_invalidate(ended=T)` immediately followed by `kg_add(valid_from=T)` at the\nidentical boundary instant -- the exact pattern MemPalace's own pre-fix agent guidance told every\ncaller to do -- and an `as_of=T` query returns both the just-ended fact and its just-started\nsuccessor at once, so a single-valued fact reports two contradictory answers with no error.\n`src/memtrust/evals/temporal_kg_boundary.py` reproduces that exact hand-rolled sequence against\n`MemPalaceAdapter`'s `kg_add()`/`kg_invalidate()`/`kg_query()` and classifies the result with a new\n`TemporalBoundarySignal` taxonomy, distinct from `ConflictSignal` and `RankingSignal` because it\nconcerns one narrow, structurally different failure: two facts sharing one instant, not a\ncontradiction across time or a ranking-order question.\n\nHonest scope, stated the same way this project states it for every other eval: the real\n`mempalace` PyPI package is not installed in this build environment, and PR#1914's fix had not\nshipped in a released `mempalace` version as of this adapter's live-verified 3.5.0 build -- it\nlands under the package's `[Unreleased]` changelog section. `tests/test_temporal_kg_boundary.py`\nproves the *classification logic* is correct against two hand-written fake implementations that\nreproduce the confirmed pre-#1914 (closed-interval) and post-#1914 (half-open-interval) SQL\ncomparison exactly. **This has not been run against a live MemPalace instance.** It is wired into\n`memtrust run --eval temporal_kg_boundary` (see \"Commands\" above); against any backend other than\n`mempalace`, it reports `not_applicable` rather than an error.\n\n![Terminal recording of the temporal-KG boundary test suite running against the hand-written pre-#1914 and post-#1914 fakes, classifying the closed-interval boundary bug and confirming the fix's half-open-interval behavior.](https://raw.githubusercontent.com/RudrenduPaul/memtrust/main/docs/assets/dev-to-demos/demo-4-temporal-kg-tests.gif)\n\n## Benchmarks\n\n**Live results: mem0_direct (self-hosted), July 2026.** MemPalace, Zep, and OpenViking are still\nnot yet measured against a live backend -- see \"Backend coverage\" below for the confidence level\non each adapter. Mem0 has one real result, produced against the actual `mem0ai` OSS library\nrunning self-hosted -- in-process, via `Mem0DirectAdapter`, backed by a local Qdrant instance and\nthe OpenAI API for embeddings and extraction.\n\n> [!NOTE]\n> This result is from the self-hosted `mem0ai` OSS library, not Mem0's hosted Platform API. Don't\n> read it as a claim about the hosted product.\n\n```\n$ export MEM0_DIRECT_EMBEDDER_PROVIDER=openai\n$ export MEM0_DIRECT_VECTOR_STORE_PROVIDER=qdrant\n$ export MEM0_DIRECT_VECTOR_STORE_URL=http://localhost:6333\n$ memtrust run --backends mem0_direct --eval contradiction,compression,extraction_quality\nmemtrust 0.3.2 -- run_id=mt_2026-07-20T210918Z\nBackends: mem0_direct Evals: contradiction, compression, extraction_quality\n\nmem0_direct: configured, running evals...\n Running Contradiction-Detection against mem0_direct...\n flagged: 0.0% silent-overwrite: 100.0% served-stale: 0.0% empty-or-lost: 0.0%\n Running Compression/Round-Trip-Fidelity against mem0_direct...\n fidelity by mode -- default: 31.6%\n Running Extraction-Quality against mem0_direct...\n junk-retained: 0.0% valid-lost: 100.0%\n feedback-loop-duplicate: 0.0%\n\nCost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)\n```\n\nThe full raw report is committed at `leaderboard/mem0_direct-2026-07-20.json`, and `leaderboard/data.json`\ncarries the contradiction numbers into the static leaderboard site (`mempalace`/`mem0`/`zep`/`openviking`\nstill show `not_measured` there; `mem0_direct` is the one real row).\n\nWhat that means case by case, not just the percentage:\n\n- **Contradiction detection, 7/7 cases: every contradicting fact silently overwrote the old one.**\n 0% were flagged as a conflict, 0% served stale, 0% empty-or-lost. Tell it your meeting moved from\n 2pm to 3pm and it stores the new fact with no signal that anything changed -- this is exactly the\n question LongMemEval and LoCoMo don't test, and exactly what \"Why this exists\" above is about.\n- **Compression/round-trip fidelity, 5 cases: 31.6% mean literal character-level reconstruction.**\n This is expected, not a defect -- mem0's design goal is semantic fact extraction, not verbatim\n storage, so a literal-reconstruction score was never going to be high. It quantifies what \"not\n built for lossless storage\" concretely means for this backend: ask it what you said and you get\n the gist back, not your words.\n- **Extraction quality, 15 cases (12 deliberately junk, 3 deliberately valid): 0% junk retained,\n 100% of the valid cases lost.** All 12 junk inputs (boot-file restating, cron heartbeat noise,\n system dumps, hallucinated-profile bait) were correctly rejected. All 3 valid-content cases were\n also dropped -- stored but never came back on retrieval. The valid-side sample is small (n=3);\n treat this as a signal worth digging into further, not a settled number.\n\n> [!WARNING]\n> A real bug this run surfaced in mem0ai itself, not in memtrust: a fresh `mem0ai==2.0.12` install\n> with nothing but `OPENAI_API_KEY` set fails every single LLM-based extraction call, out of the\n> box, for anyone.\n\nGetting any of the numbers above required a fix first. mem0's own default model\n(`mem0/llms/openai.py`: `self.config.model = \"gpt-5-mini\"`) is a reasoning-tier model that only\naccepts the API's default temperature, but mem0's own reasoning-model detection\n(`mem0/llms/base.py`'s `reasoning_models` set) checks for the string `\"gpt-5o-mini\"`, not\n`\"gpt-5-mini\"` -- two different strings, so the check never fires, and mem0 sends `temperature=0.1`\non every call regardless. The result is a `400 Unsupported value: 'temperature' does not support\n0.1 with this model` error on every extraction call, silently caught by mem0 and reported by\nmemtrust as `N/A (no scoreable cases)` rather than a real result. `Mem0DirectAdapter` now works\naround it by passing `is_reasoning_model=True` explicitly -- mem0's own documented override for\nexactly this situation -- see `src/memtrust/adapters/mem0_direct_adapter.py`'s \"Default LLM\nextraction is broken out of the box\" section for the full citation. No upstream mem0ai issue filed\nfor this as of this writing.\n\nTo reproduce this or measure the remaining three backends:\n\n```bash\nexport MEM0_API_KEY=... # and/or\nexport ZEP_API_KEY=...\nexport OPENVIKING_API_KEY=...\nexport MEMPALACE_STORAGE_PATH=...\nexport MEMTRUST_JUDGE_API_KEY=... # needed for LongMemEval/LoCoMo grading; contradiction-detection doesn't need it\n\nmemtrust run --backends mempalace,mem0,zep,openviking --eval all\nmemtrust report memtrust-report-.json\n```\n\nThe command prints per-backend accuracy and contradiction-handling rates, writes a full JSON\nreport, and prints an estimated cost for any LLM-judged evals that ran. `MemPalaceAdapter`'s\ndrawer and knowledge-graph calls are now live-verified against a real installed instance (see\n\"Backend coverage\" below), but OpenViking's memory-write/query paths, and parts of the\nself-hosted Mem0 and Zep/Graphiti adapters, are still built against best-effort interpretations of\ndocumented or source-read product concepts rather than a live-confirmed API -- see the confidence\ntable in `docs/methodology.md` before treating any adapter's output as authoritative, and consider\nthat table's gaps a standing invitation to contribute a fix.\n\n**Labeling requirement for any future `accuracy` figure published here.** LongMemEval and LoCoMo\n`accuracy` grades the LLM judge's verdict on raw retrieved-record content directly -- there is no\nanswer-generation step in either eval runner. This is not the same measurement as the official\nLongMemEval/LoCoMo leaderboards' generate-then-judge QA-accuracy scores. Any `accuracy` number\nthis project publishes for those two evals must be labeled \"retrieval-graded accuracy,\" not bare\n\"accuracy,\" and must not be directly compared to leaderboard figures without that caveat. See\n`docs/methodology.md`'s \"Retrieval-graded accuracy vs. generated-answer accuracy\" section.\n\n## GitHub Actions usage\n\nRun the suite on a schedule and publish results to the leaderboard:\n\n```yaml\nname: memtrust-leaderboard\non:\n schedule:\n - cron: \"0 9 * * 1\" # weekly\n workflow_dispatch: {}\n\njobs:\n benchmark:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - uses: actions/setup-python@v5\n with:\n python-version: \"3.12\"\n - run: pip install memtrust-cli\n - run: memtrust run --backends mempalace,mem0,zep,openviking --eval all --output leaderboard/data.json\n env:\n MEM0_API_KEY: ${{ secrets.MEM0_API_KEY }}\n ZEP_API_KEY: ${{ secrets.ZEP_API_KEY }}\n OPENVIKING_API_KEY: ${{ secrets.OPENVIKING_API_KEY }}\n MEMTRUST_JUDGE_API_KEY: ${{ secrets.MEMTRUST_JUDGE_API_KEY }}\n - run: git add leaderboard/data.json && git commit -m \"Update leaderboard\" && git push\n```\n\nThis repo's own CI (`.github/workflows/ci.yml`) runs lint, type-check, test, and a dependency\nsecurity audit on every push and pull request -- no vendor credentials required, since every test\nruns fully offline.\n\n## Self-host\n\n```bash\ngit clone https://github.com/RudrenduPaul/memtrust\ncd memtrust\npip install -e \".[dev]\"\nexport MEM0_API_KEY=...\nmemtrust run --backends mem0 --eval all\n```\n\nPoint an adapter at your own backend, or run the suite against your own conversation data instead\nof the bundled synthetic fixtures (see `docs/methodology.md`'s note on swapping in the real\nLongMemEval/LoCoMo datasets). Nothing leaves your machine unless you choose to publish it.\n\n## Install\n\n`pip install memtrust-cli` is the verified, working install path -- confirmed against a clean\nvirtualenv as of this writing. `pip show memtrust-cli` and `memtrust --version` both report\n`0.3.4`.\n\n### npx (currently broken -- tracked, not hidden)\n\nThe npm package (`memtrust-cli`) is live and no longer 404s, and its source on `main`\n(`npm/memtrust-cli/bin/memtrust.js`) correctly runs `uv tool run --from\nmemtrust-cli== memtrust `. The published `0.3.4` npm tarball, however, still\nships the earlier, broken build of that same file, which runs `uv tool run --from\nmemtrust== memtrust ` instead -- pointed at a PyPI project named `memtrust` that\nhas never existed (`pypi.org/pypi/memtrust/json` returns 404, same as the FAQ below already\ndocuments). The source fix landed on `main`; the npm publish that would ship it has not gone\nout yet. Confirmed live, today, by downloading the actual published tarball\n(`npm pack memtrust-cli@0.3.4`) and inspecting `bin/memtrust.js` directly, not by reading source\nand assuming it matches what's published:\n\n```bash\n$ npx -y memtrust-cli --version\nnpm error could not determine executable to run\n × No solution found when resolving tool dependencies:\n ╰─▶ Because memtrust was not found in the package registry and you\n require memtrust==0.3.4, we can conclude that your requirements are\n unsatisfiable.\n```\n\nUntil a new npm version ships with the fixed wrapper, use `pip install memtrust-cli` (above) --\nit is unaffected, since the bug is only in the npm wrapper script, not the PyPI package it\nbootstraps. For CI and agent runners that have Node.js but not Python: hold off on `npx\nmemtrust-cli` until this section no longer carries this notice, or provision Python and use `pip\ninstall memtrust-cli` directly.\n\nThe npm package is named `memtrust-cli` so it is unambiguous as a CLI tool at a glance (and so it\ndoesn't collide with any future `memtrust` JS library package). `npx` always resolves the package\nname to its matching `bin` entry automatically, so `npx memtrust-cli ...` is the intended\nzero-install path once the fixed build ships. Once installed, the package also exposes the\nshorter `memtrust` command as a second `bin` alias -- matching the underlying Python CLI's own\ncommand name -- so you are not stuck typing `memtrust-cli` for every subsequent invocation.\n\nThis was never meant to be a zero-dependency install: `npx memtrust-cli` still fetches\n`memtrust-cli` from PyPI on first use. What it removes is a Python toolchain to provision by\nhand -- it bootstraps the interpreter and package fetch for you via a bundled, verified copy of\nAstral's [`uv`](https://github.com/astral-sh/uv). Each platform package bundles a genuine,\nSHA-256-verified copy of `uv`'s own GitHub release binary (fetched at npm package-publish time,\nnever at end-user install time). The npm package is pinned to its own version -- bump\n`npm/memtrust-cli/package.json`'s version and republish when a new PyPI release ships, and every\nsubsequent install resolves to that exact release, not whatever happens to be newest at run\ntime. That republish is exactly the step still outstanding here.\n\n## What a hosted trust layer would add\n\nThe harness, adapters, and leaderboard in this repo are the entire OSS surface, and they're\nsufficient on their own to compare backends. A hosted layer on top of this -- described here, not\nbuilt -- would add continuous regression monitoring that re-runs the suite automatically whenever\na tracked backend ships a new release, private scorecards that run the same methodology against a\nteam's own data shape instead of the public sample fixtures, and a compliance-report export for\nteams whose security or legal review needs a documented third-party artifact rather than a\nfree-text summary. None of that exists yet. If it's ever built, it stays additive to the free\nharness, never a requirement for using it.\n\n## Backend coverage\n\nThe MemPalace row below used to say \"needs verification against a live instance\" -- it needed more\nthan that. Every prior version of `MemPalaceAdapter` called a `mempalace.Palace` class\n(`Palace(storage_path=...)` exposing `.remember()`/`.recall()`/`.invalidate()`) that never existed\nin the real, installed package. `python3 -c \"import mempalace; hasattr(mempalace, 'Palace')\"`\nreturns `False`; grepping every `class` definition across the installed package turns up nothing\nnamed `Palace` anywhere. Every test that appeared to pass before this rewrite was exercising a\nhand-written fake standing in for that guess, never the real thing -- `store()`/`query()`/\n`update()` had never actually worked against a live MemPalace install, in this project's entire\nhistory, until this rewrite. `src/memtrust/adapters/mempalace_adapter.py` was rewritten from\nscratch against the real, plain module-level functions in `mempalace.mcp_server`\n(`tool_add_drawer`, `tool_search`, `tool_update_drawer`, `tool_delete_drawer`,\n`tool_kg_add`/`tool_kg_invalidate`/`tool_kg_query`) -- every return shape documented in the\nadapter's module docstring was captured by calling those functions live against a real, local\nchromadb-backed palace, not read off a docstring and trusted. It's the kind of mistake this whole\nproject exists to catch in other people's benchmarks; finding it in memtrust's own adapter and\nshipping the fix in the open, rather than quietly patching it, is the more useful story.\n\n![Terminal recording discovering that mempalace.Palace never existed in the installed package and walking the real mempalace.mcp_server functions that MemPalaceAdapter now calls instead.](https://raw.githubusercontent.com/RudrenduPaul/memtrust/main/docs/assets/dev-to-demos/demo-2-fictional-api-discovery.gif)\n\n| Backend | Adapter status | Confidence (see docs/methodology.md) |\n|---|---|---|\n| MemPalace | Implemented -- drawer API + knowledge-graph API | High on the real `mempalace.mcp_server` functions this adapter now calls, live-verified against an installed `mempalace` 3.5.0 instance (see above). Still best-effort on compression-mode names (`\"raw\"`/`\"AAAK\"`) and on whether `degraded_retrieval` warnings are ever populated by the installed version -- see the adapter's module docstring for both caveats stated plainly. |\n| Mem0 | Implemented -- hosted Platform API, self-hosted OSS server, and a direct in-process library adapter | High on the hosted Platform API and on what the installed `mem0ai==2.0.12` library's embedder/vector-store code actually does (confirmed by reading its real source, exercised directly in tests); medium-high on the self-hosted OSS server's route shape (confirmed from source, not run against a live server). |\n| Zep / Graphiti | Implemented -- hosted Zep Platform API and a self-hosted `graphiti-core` adapter | Medium-high on the hosted API's documented contradiction-handling behavior; medium on the self-hosted adapter's wire-level shape (every method signature confirmed by reading `graphiti-core`'s real source, not by running it against a live Neo4j/FalkorDB instance -- the package isn't installed in this environment). |\n| OpenViking | Implemented | Medium on architecture, low on exact memory-write/query paths -- still the adapter most likely to need correction against a live instance. |\n\nAdding a backend adapter is the primary contribution path -- see `CONTRIBUTING.md`.\n\n## Development\n\n```bash\npip install -e \".[dev]\"\nruff check . && ruff format --check .\nmypy --strict src/memtrust\npytest --cov=memtrust --cov-report=term-missing --cov-fail-under=80\npip-audit\n```\n\n`.pre-commit-config.yaml` wires ruff and mypy into `pre-commit` if you'd rather run these on every\ncommit than remember to run them by hand.\n\n## FAQ\n\n**What is memtrust, and what actually makes it different from reading a vendor's own benchmark\npage?** It's a CLI harness that runs the same evals (LongMemEval, LoCoMo, and 15 others registered\nin `--eval`, including a contradiction-detection eval none of the four tracked backends publish a\nnumber for) against MemPalace, Mem0, Zep/Graphiti, and OpenViking, and prints the raw output rather\nthan a curated summary. The differentiator isn't a proprietary scoring model; it's that nobody\noutside the vendor had previously run the same test, the same way, against every option, with the\nfull methodology published alongside the code that produced it (`docs/methodology.md`). See \"Why\nthis exists\" above for the MemPalace LongMemEval overclaim (mempalace/mempalace#27) that motivated\nthe project.\n\n**Does memtrust support my platform, and what happens if it doesn't?** The npm wrapper\n(`memtrust-cli`) ships six platform-specific optional-dependency packages --\n`@memtrust-cli/darwin-x64`, `@memtrust-cli/darwin-arm64`, `@memtrust-cli/linux-x64`,\n`@memtrust-cli/linux-arm64`, `@memtrust-cli/win32-x64`, and `@memtrust-cli/win32-arm64` -- each\nbundling a verified `uv` binary for that exact platform (`npm/memtrust-cli/bin/memtrust.js`).\n`npm install` picks whichever one matches `process.platform`/`process.arch` at install time. On an\nunsupported combination (32-bit x86, or any platform outside that list), the wrapper exits with a\nclear `no prebuilt uv binary available for /` error instead of a silent failure. The\nunderlying `memtrust` PyPI package itself only requires Python 3.11+, so `pip install memtrust-cli`\nremains the fallback path on any platform the npm wrapper doesn't cover.\n\n**Why does `npx memtrust-cli --version` fail with a \"memtrust was not found in the package\nregistry\" error?** A real, currently-live packaging gap, confirmed by downloading the published\ntarball directly (`npm pack memtrust-cli@0.3.4`) rather than trusting the repo's source: the\n`0.3.4` build on npm still runs `uv tool run --from memtrust==`, pointed at a PyPI\nproject named `memtrust` that has never been published. The fix (`--from\nmemtrust-cli==`, the real published name) is already merged on `main`; it just hasn't\ngone out in an npm release yet. `pip install memtrust-cli` is unaffected and is the reliable\ninstall path until that release ships -- see \"Install\" above for the exact reproduction.\n\n**Do I need a Python toolchain installed to use memtrust?** The npm wrapper is designed to make\nthat unnecessary -- `npx memtrust-cli run ...` is meant to run `uv tool run --from\nmemtrust-cli== memtrust ` under the hood, letting `uv` provision its own\nisolated Python interpreter on first use. As of this writing the published `0.3.4` npm package\nstill ships an earlier, broken build of that wrapper (see \"Install\" above for the confirmed\nrepro); until a fixed version is published, you do need Python 3.11+ installed and should use\n`pip install memtrust-cli` (the PyPI package name) directly, no Node.js involved.\n\n**How does memtrust compare to a general-purpose LLM eval framework like RAGAS?** RAGAS evaluates\nRAG pipelines and other LLM applications with objective metrics and synthetic test-data generation;\nit has no memory-backend adapter abstraction and no eval built around a fact contradicting an\nearlier one, because that's not the problem it's built to solve. memtrust is narrower on purpose: it\nonly tracks four named agent-memory backends and its non-recall evals (contradiction detection,\ncompression/round-trip fidelity, temporal-KG boundary detection) exist specifically to test claims\nthose four backends make about themselves. If you need broad RAG or prompt-evaluation coverage,\nRAGAS or a similar framework is the right tool; if you need to check whether a memory backend\nsilently drops or overwrites a contradicted fact, memtrust is the one built for that question.\n\n**Why did `memtrust --version` used to print a version that didn't match what pip said I\ninstalled?** This was a real, shipped bug in an early release, not a hypothetical one: installing\n`memtrust-cli` from PyPI into a clean virtualenv and running `pip show memtrust-cli` reported the\ncorrect version, but `memtrust --version` printed `0.0.0+unknown` regardless, because\n`src/memtrust/__init__.py` read `importlib.metadata.version(\"memtrust\")` -- the wrong distribution\nname -- instead of `version(\"memtrust-cli\")`, the name the package is actually installed under.\nThe first attempted fix was itself incomplete: it hardcoded the lookup to `\"memtrust-cli\"`, which\nwould have broken a `memtrust`-named mirror install the same way in reverse -- an environment with\nonly a `memtrust`-named distribution installed has no `memtrust-cli` entry in its own\ninstalled-package metadata at all, so the lookup would always miss and fall through to the same\n`0.0.0+unknown` fallback. Fixed for real in the following release: the lookup now tries\n`memtrust-cli` first, falls back to `memtrust`, and only reports `0.0.0+unknown` if neither\ndistribution name is installed. That fallback is defensive, not evidence a `memtrust`-named PyPI\nproject exists today -- see \"Can I `pip install memtrust` instead of `memtrust-cli`?\" below for\nthe current, corrected answer.\n\n**Can I `pip install memtrust` instead of `memtrust-cli`?** No, not currently -- this README\npreviously claimed a separate `memtrust`-named PyPI project existed in sync with `memtrust-cli`.\nThat was checked live against `pypi.org/pypi/memtrust/json` and it returns a plain 404: no project\nnamed `memtrust` has ever been published. `pip install memtrust` fails with \"No matching\ndistribution found for memtrust.\" `memtrust-cli` is the one real, published PyPI name; use that.\n(`src/memtrust/__init__.py` still has a defensive fallback that would also read a `memtrust`-named\ndistribution's version if one were ever installed locally -- e.g. from a local build -- but that\nis unrelated to whether a `memtrust` project is live on PyPI, which it is not.) The npm wrapper's\n`bin/memtrust.js` on `main` now pins `uv tool run --from memtrust-cli==` for the same\nreason -- the published `0.3.4` npm build has not picked that fix up yet, though; see \"Install\"\nabove for the current, confirmed-broken state of `npx memtrust-cli`.\n\n**Has memtrust actually been run against a live memory backend, or is this all synthetic?** Both,\nand the README doesn't blur the line. The eval logic itself is proven against bundled synthetic\nfixtures and, for several adapters, real installed vendor packages with only the network boundary\nmocked (see the pytest coverage table above). One backend has a real live result: `mem0_direct`,\nthe self-hosted `mem0ai` OSS library (not Mem0's hosted Platform API), run against contradiction,\ncompression, and extraction-quality -- see \"Benchmarks\" above for the exact numbers and a real bug\nthat run surfaced in `mem0ai` itself. MemPalace, the hosted Mem0 Platform API, Zep, and OpenViking\nhave not yet been run against a live backend with real credentials as of this writing. The\n\"Backend coverage\" table gives a per-adapter confidence level (high/medium/low) for exactly this\nreason; run it yourself against your own credentials to get a live-verified number.\n\n**Can I use memtrust commercially, and does it require attribution?** Yes. It's licensed under\nApache License 2.0 (see `LICENSE`), which permits commercial use, modification, and distribution,\nand requires you to preserve the license and copyright notice and to note any changes you make to\nthe code. It does not require you to open-source your own product just because you depend on\nmemtrust.\n\n**How do I get real accuracy numbers for the backends still marked \"not yet measured\"?** Set the\ncredential environment variable for whichever backend you want to test\n(`MEMPALACE_STORAGE_PATH`, `MEM0_API_KEY`, `ZEP_API_KEY`, `OPENVIKING_API_KEY`, plus\n`MEMTRUST_JUDGE_API_KEY` for LLM-judged evals like LongMemEval and LoCoMo), then run\n`memtrust run --backends --eval all` and `memtrust report `. A backend with no\ncredential configured prints `SKIPPED` and the run still completes and writes a valid JSON report --\nsee \"What it does\" and \"Benchmarks\" above for the exact commands.\n\n## License\n\nApache 2.0. See `LICENSE`.\n\n## Success stories\n\n197 real issues/PRs filed by real contributors against MemPalace, mem0, Zep/Graphiti, and\nOpenViking have been independently root-caused against this codebase: does the solution, as it\nactually exists today, let you diagnose or resolve what was reported? 55 (28%) verify as a clean\nPASS, 16 (8%) as PARTIAL (evidence captured, needs a human to\ninterpret further, or only part of the issue is covered), 42 (21%) as a genuine capability gap this\nharness doesn't close yet, and 84 (43%) as not actually applicable (feature requests,\nalready-fixed-upstream, or genuinely out of scope). Every verdict below has been re-verified live\nagainst the current codebase by a reviewer independent of whoever built the fix, not just cited\nfrom a changelog. Full write-ups and validation evidence are tracked internally; the summary here\nis for anyone deciding whether this harness would have caught their own bug.\n\n**The headline story is about memtrust's own bug, not a vendor's.** Every version of\n`MemPalaceAdapter` before this rewrite called a `mempalace.Palace` class -- `Palace(storage_path=\n...)` exposing `.remember()`/`.recall()`/`.invalidate()` -- that never existed in the real,\ninstalled package. `python3 -c \"import mempalace; hasattr(mempalace, 'Palace')\"` returns `False`;\nnothing named `Palace` appears anywhere in the installed package's source. Every test that appeared\nto pass was exercising a hand-written fake standing in for that guess -- `store()`/`query()`/\n`update()` had never once worked against a live MemPalace install. `src/memtrust/adapters/\nmempalace_adapter.py` was rewritten against the real `mempalace.mcp_server` functions, with every\ndocumented return shape captured by calling them live against a real local instance. A project\nbuilt to catch other vendors overclaiming found the same failure mode in its own code, and the fix\nshipped in the open rather than quietly. See \"Backend coverage\" above for the full account.\n\n**MemPalace**\n- [#1754](https://github.com/MemPalace/mempalace/pull/1754) (@rodboev): a checkpoint recovery fix\n for silently quarantined dim-None pickles. memtrust's contradiction eval couldn't previously tell\n \"silently quarantined\" apart from \"no update primitive at all\"; it now can\n (`ConflictSignal.EMPTY_OR_LOST`).\n- [#1929](https://github.com/MemPalace/mempalace/pull/1929) (@jrzmurray): a fix for NUL bytes\n silently corrupting a ChromaDB index. memtrust's `store()` used to trust \"no exception\" as proof\n of a durable write; an opt-in read-after-write verification step now catches this.\n- [#1450](https://github.com/MemPalace/mempalace/pull/1450) (@lealbrunocalhau): a fix for an empty\n embedding response getting scored as a wrong answer instead of flagged as infra failure. Same\n fix as #1754 above.\n- [#1823](https://github.com/MemPalace/mempalace/pull/1823) / [#1543](https://github.com/MemPalace/mempalace/pull/1543)\n (@fatkobra): lock and write-integrity fixes that pointed at the same read-after-write gap #1929\n closed.\n- [#1913](https://github.com/MemPalace/mempalace/issues/1913) / [PR#1914](https://github.com/MemPalace/mempalace/pull/1914)\n (@ggettert): a temporal-KG `as_of` boundary bug where a fact ending at exactly the query instant\n still matched alongside its successor. memtrust's new temporal-KG boundary eval reproduces the\n exact hand-rolled `kg_invalidate()`-then-`kg_add()` sequence that triggers it -- see \"The eval\n built from MemPalace's own bug\" above for the honest not-yet-live-verified caveat.\n- [PR#1890](https://github.com/MemPalace/mempalace/pull/1890) / [#1889](https://github.com/MemPalace/mempalace/issues/1889)\n (@JosefAschauer): an `authored_at` chronology tie-break fix for `_hybrid_rank`. memtrust's ranking\n classifier now credits a top-level `authored_at` field, not just one nested under `metadata`, as\n a genuine ranking-driving signal.\n\n**mem0**\n- [#5973](https://github.com/mem0ai/mem0/pull/5973) (@abhay-codes07, superseded by\n [#5992](https://github.com/mem0ai/mem0/pull/5992)): an empty-string entity-id filter scoping bug.\n memtrust's mem0 adapter only reached the hosted Platform API and had no delete operation at all,\n so it couldn't have caught this. A self-hosted adapter with tested delete/delete_many primitives\n now can.\n- [#4297](https://github.com/mem0ai/mem0/pull/4297) (@utkarsh240799): a dimension auto-detection\n fix. The self-hosted adapter now routes to the right deployment, though no test yet reproduces\n this specific bug end to end, so this one is partial, not fully caught.\n- [#4573](https://github.com/mem0ai/mem0/issues/4573) (@jamebobob): a 32-day audit of 10,134 real\n mem0 entries finding 97.8% junk. memtrust's new extraction-quality eval and\n `ExtractionQualitySignal` taxonomy cover the audit's own junk categories, including its\n 808-duplicate feedback-loop case.\n- [PR#5980](https://github.com/mem0ai/mem0/pull/5980) (@HrushiYadav): a filter-injection fix for\n the Elasticsearch vector store. A new filter-injection eval exercises the real, installed\n `mem0.vector_stores.elasticsearch.ElasticsearchDB._validate_filter()` directly and confirms it\n rejects the exact malicious filter shape (`{\"user_id\": {\"$ne\": \"\"}}`) this PR fixed.\n- [#4956](https://github.com/mem0ai/mem0/issues/4956) (@NDNM1408): an open proposal that mem0's\n add-only pipeline surfaces stale, contradictory facts with no recency signal. memtrust's\n contradiction eval now runs the same literal add-only scenario (two `store()` calls, no explicit\n update) against this taxonomy.\n- [#4884](https://github.com/mem0ai/mem0/issues/4884) (@wangjiawei-vegetable): a hardcoded\n English-only tokenizer silently degrading non-Latin-script retrieval. A new\n `LanguageDegradationSignal` and non-Latin-script fixtures now catch this shape --\n `Mem0DirectAdapter`-specific (it reads `query(explain=True)`'s real per-result diagnostic\n fields, a capability only that adapter exposes) and, like `embedder_cost.py`'s cost-attribution\n eval and `episode_temporal_leak.py`'s Graphiti-specific eval, not yet wired into `memtrust run\n --eval`'s general list; call `run_language_degradation_eval()` directly, or see the\n `test_query_language_degradation_*` tests in `tests/test_mem0_direct_adapter.py`, until that CLI\n surface exists.\n\n**Zep / Graphiti**\n- [#1489](https://github.com/getzep/graphiti/issues/1489) (@brentkearney): a bi-temporal\n `invalid_at` correctness gap. memtrust's contradiction classifier used to discard Graphiti's own\n `invalid_at` metadata and infer everything from a fixed top-5 text match, misreading a correctly\n flagged case as a silent overwrite. It now checks the metadata first.\n- [#1275](https://github.com/getzep/graphiti/issues/1275) (@rafaelreis-r, still open): O(n)\n entity-resolution context growth silently dropping episodes past roughly 300 ingested. A new\n self-hosted `graphiti-core` adapter plus a scale/volume-stress eval now tracks a fixed \"anchor\"\n record's recall across ascending checkpoints against real `add_episode()` ingestion -- the same\n shape this issue describes.\n- [#836](https://github.com/getzep/graphiti/issues/836) (@matthiaslau) / [#920](https://github.com/getzep/graphiti/issues/920)\n (@markwkiehl): two separate crashes in `update_communities()`/`resolve_edge_contradictions()` --\n a too-many-values-to-unpack error and a tz-naive/aware datetime comparison error. A new\n `CrashSignal` classification recognizes both exact shapes instead of surfacing an opaque generic\n exception.\n- [PR#1222](https://github.com/getzep/graphiti/pull/1222) (@david-morales) / [PR#1183](https://github.com/getzep/graphiti/pull/1183)\n (@Milofax): FalkorDB RediSearch syntax errors from empty or unescaped fulltext queries. A new\n `CrashSignal.QUERY_SANITIZATION_ERROR` recognizes both issues' verbatim filed error text.\n- [#1467](https://github.com/getzep/graphiti/issues/1467) (@elimydlarz, open, zero engagement):\n `GeminiEmbedder` silently returning the wrong vector count. A new\n `CrashSignal.EMBEDDING_BATCH_COUNT_MISMATCH` catches this once Gemini embedder support is wired\n into the self-hosted adapter.\n\n**OpenViking**\n- [#3029](https://github.com/volcengine/OpenViking/issues/3029) (@dfwgj, still open): Feishu resync\n silently deleting user-managed files. memtrust had no way to observe this failure mode at all; a\n dedicated resource-sync-safety eval now seeds generated and user files, triggers a resync, and\n checks what survives.\n- [#2850](https://github.com/volcengine/OpenViking/issues/2850) (@lg320531124, still open): BM25\n search silently returning empty results at scale. A dedicated scale/volume-stress eval\n (`memtrust run --eval scale_stress`) now stores a large synthetic corpus and re-queries it at\n ascending checkpoints to reproduce the *shape* of this condition -- recall collapsing past a\n volume threshold with no exception raised.\n- [#1581](https://github.com/volcengine/OpenViking/issues/1581) (@0xble, fix rejected, still live\n upstream): `v2_lock_max_retries=0` silently means unlimited retries, not zero. A new\n lock-contention eval asserts a bounded response-time budget under concurrent-write contention.\n- [#1255](https://github.com/volcengine/OpenViking/issues/1255) (@SeeYangZhi): a stats endpoint\n silently returning zero despite persisted memories. A new `get_stats()`/`StatsResult` primitive\n and dedicated stats-accuracy eval now catch this.\n- [#2966](https://github.com/volcengine/OpenViking/issues/2966) (@lRoccoon, unaddressed upstream):\n legacy uint16-truncated records that are permanently undeletable. A new\n `CrashSignal.LEGACY_CORRUPT_RECORD_UNDELETABLE` now surfaces this instead of a silent no-op.\n- [#204](https://github.com/volcengine/OpenViking/issues/204) (@ponsde, closed): non-deterministic\n search results (Jaccard similarity 0.11 across identical queries) from a self-diagnosed dimension\n mismatch. A new result-consistency eval computes pairwise Jaccard similarity over repeated\n identical queries to catch this class directly.\n\n**Cross-project**\n- [OneNomad-LLC/przm-bench](https://github.com/OneNomad-LLC/przm-bench) (@mattstvartak): a peer\n benchmarking project shipped cryptographic receipt signing; memtrust had none. `memtrust` now has\n real Ed25519 signing/verification (the `cryptography` library, not a hand-rolled scheme) via\n `memtrust keygen` / `run --sign` / `verify` -- a tampered receipt correctly fails verification,\n a genuine one correctly passes.\n\nSeveral PARTIAL and FAIL -- capability gap rows above and elsewhere in the full 197-row set remain\nopen, deliberately not counted as fixed: some point at real gaps this harness genuinely can't close\nyet without a live vendor credential, and inflating a near-miss to PASS defeats the entire point of\nan independently-verified benchmark. See the confidence caveats throughout this README and in\n`docs/methodology.md` for exactly which claims rest on which kind of evidence.\n","changelogMarkdown","videoKey","requiredSecretsJson","publishedAt","priceCents","trialDays",0,"upstream",{"_108":109,"_110":111,"_112":113,"_114":115,"_116":65,"_117":118},"requiredSecrets",[],"tags",[93,94,95,96,97,98,99,100,101,102,103,104,105,106,107],"assets",[],"stats",{"_91":80,"_92":80},"ratingAvg","reviewCount","agent-memory","ai-agents","benchmark","cli","contradiction-detection","eval","graphiti","llm","llm-eval","locomo","longmemeval","mem0","mempalace","python","zep","topics",[93,94,95,96,97,98,99,100,101,102,103,104,105,106,107],"license","Apache-2.0","language","Python","registryName","io.github.RudrenduPaul/memtrust","homepage","packages",[119],{"_120":121,"_122":123,"_70":71},"registryType","pypi","identifier","memtrust-cli","ENV",{"_126":127,"_128":129,"_130":131},"ENVIRONMENT","production","TURNSTILE_SITE_KEY","0x4AAAAAAB8-rEI2gNDf_QhU","SENTRY_DSN","https://45fd1229f78089d5c3ee78be128030e1@o4510583261757440.ingest.de.sentry.io/4510584253448272"]