Reproducible benchmark results.

This page renders results from github.com/sovantica/engrava-benchmark, a public, MIT-licensed reproduction of standard agent-memory benchmarks. Numbers here are read from that repository at a pinned commit at build time — there is no live API and no client-side fetch on this page.

Scores are only comparable to each other within the same segment below — the same benchmark, dataset revision, split, harness, reader, judge, and scorer. Different segments are never merged into one ranked table, because changing any of those axes changes what the number means.

No language model runs inside the memory pipeline itself — retrieval is embeddings only, and the reader and judge that turn retrieved context into a graded answer sit outside it. Every verified row was produced by the public engrava package under the official scorer, and carries its own audit trail: hypotheses, judge labels and a checksummed artifact, committed in the repository alongside the row. That is what lets a third party check a score without re-running it.

Engrava 0.5.0, 0.6.0 and 0.7.0 sit in separate segments because they ran on different harness commits, the only axis separating them. Of five hundred graded questions, 0.5.0 answered 412 correctly, 0.6.0 answered 408 and 0.7.0 answered 414. 0.6.0 is the lowest of the three and 0.7.0 the highest, and all three are within six questions of each other. Judge verdicts carry non-zero error, and the benchmark methodology treats small gaps as noise even inside a single segment; these rows are not in one. Neither an improvement nor a regression is demonstrated.

For 0.7.0 against 0.6.0 the two runs can be compared prompt by prompt, because the reader prompt is built from what Engrava retrieved. 493 of the 500 reader prompts are byte-identical, and all 26 questions whose outcome changed are among them, so every observed change in outcome came with an identical reader input. The 0.7.0 score does not establish an improvement over 0.6.0. The pull request that added the 0.7.0 row has the comparison.