benchmark stringclasses 1
value | sessions_on_disk int64 19.2k 19.2k | questions int64 100 100 | corpus_source stringclasses 1
value | tool stringclasses 7
values | version stringclasses 7
values | reads_existing_history bool 2
classes | agent_formats_supported stringclasses 7
values | install_size stringclasses 7
values | background_process stringclasses 6
values | needs_model_or_key bool 2
classes | mcp_schema_tokens_o200k int64 0 8.28k | mcp_tool_count int64 0 45 | build_seconds float64 13.6 10.8k ⌀ | first_answer_ms int64 26 4.5k ⌀ | p50_search_ms int64 97 2.6k ⌀ | hit_at_1 int64 0 19 | hit_at_5 int64 0 36 | found_at_50 int64 0 65 | notes stringclasses 7
values |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Day-Zero Cold Start Comparison | 19,195 | 100 | longmemeval_s_cleaned.json | deja-vu | 0.23.2 (build) / 0.21.1 (MCP schema) | true | 48 | 18.6 MB binary | none | false | 477 | 1 | 13.6 | 26 | 97 | 19 | 36 | 65 | Build time is mean of three cold builds on 0.23.2 (13.5-13.8 s); accuracy row build took 17.6 s. MCP schema collapsed to one tool with modes. |
Day-Zero Cold Start Comparison | 19,195 | 100 | longmemeval_s_cleaned.json | ctx | 1.4.12 (setup re-tested on 1.6.3, 2.2.8) | true | 39 providers | 43 MB binary | daemon | false | 1,897 | 6 | 72 | 1,040 | 1,030 | 7 | 14 | 34 | Explicit ctx import indexed 19,195 sessions (199,499 events, 553 MB store). Default ctx setup returned 0 sessions on a single project dir; on 2.2.8 across 200 dirs it completed in 72.7 s (hit@1 7, hit@5 18, found@50 37). |
Day-Zero Cold Start Comparison | 19,195 | 100 | longmemeval_s_cleaned.json | CASS | 0.8.0 (CLI) / main 38f0412 (accuracy) | true | 25+ | 58.5 MB binary | none | false | 0 | 0 | 3,360 | 380 | 370 | 5 | 8 | 16 | CLI agent surface (no standing MCP schema overhead). Strict AND keyword semantics; evaluated with stop words stripped from queries. |
Day-Zero Cold Start Comparison | 19,195 | 100 | longmemeval_s_cleaned.json | agentsview | 0.44.0 (MCP) / 0.43.0 (accuracy) | true | 62 | 115 MB binary | daemon, for search | false | 8,283 | 7 | 236 | 1,440 | 113 | 7 | 13 | 19 | Evaluated with --fts and stop words stripped. Synced 38,390 rows across both layouts (398,998 messages, 915 MB SQLite DB). |
Day-Zero Cold Start Comparison | 19,195 | 100 | longmemeval_s_cleaned.json | agentmemory | 0.9.29 | true | Claude Code JSONL import (<=1000 files/call) | 689 MB npm + 28 MB engine | worker + engine, four ports | false | 745 | 7 | 95 | 185 | 145 | 14 | 34 | 65 | Evaluated in documented keyless mode (BM25 only). |
Day-Zero Cold Start Comparison | 19,195 | 100 | longmemeval_s_cleaned.json | MemPalace | 3.9.0 | true | file tree mining | 312 MB + 168 MB embedding model | Chroma | true | 8,183 | 45 | 10,800 | 4,500 | 2,600 | 14 | 20 | 46 | Approximately 3 hours local mining via mempalace mine --mode convos over Claude layout. |
Day-Zero Cold Start Comparison | 19,195 | 100 | longmemeval_s_cleaned.json | claude-mem | 13.25.3 | false | records forward from install | npm + Claude Code plugin | worker | true | 2,307 | 15 | null | null | null | 0 | 0 | 0 | Forward-only session-end hook recorder requiring an AI provider; holds no prior sessions on day zero. |
Coding Agent Memory Benchmark
Machine-readable summary splits generated from the verified JSON artifacts in vshulcz/coding-agent-memory-benchmark (Zenodo archive: 10.5281/zenodo.24416712) for vshulcz/deja-vu.
Disclosure: published and maintained by the author of
deja-vu. Upstream datasets carry their own licenses (LongMemEval: MIT;LoCoMo:CC-BY-NC-4.0). To respectLoCoMo's license, this dataset redistributes aggregate retrieval metrics and dataset digests only—no raw dialogue text or prompts are included.
Configs & Splits
| Config | Rows | Contents | Documentation |
|---|---|---|---|
longmemeval_slices |
14 | Per-category and total retrieval metrics (hit@1/5/10/20, MRR, latency, sha256) on cleaned (470q) and full (500q) LongMemEval-S sets (deja-vu 0.23.2) |
Benchmarks guide |
locomo_retrieval_metrics |
6 | Aggregate retrieval-only metrics over 1,982 LoCoMo questions (multi-hop, temporal, open-domain, single-hop, adversarial*, TOTAL) |
Benchmarks guide |
day_zero_tools |
7 | Head-to-head cold-start build time, query latency, hit@1/5, found@50, and standing MCP tools/list schema overhead over 19,195 on-disk sessions |
Day-zero evaluation |
task_cost_and_compaction |
10 | End-to-end repository task token/call ablations (11 runs per arm), 43-compaction retention measurements, and context-readiness baselines |
Day-zero evaluation |
Reproducing & Submitting External Rankings
Clone the benchmark repository and score any external ranking JSON with Python 3 standard library only:
git clone https://github.com/vshulcz/coding-agent-memory-benchmark
cd coding-agent-memory-benchmark
python3 scripts/verify_artifacts.py
python3 scripts/score_longmemeval.py --data /path/to/longmemeval_s_cleaned.json --score submission.json --skip-abs
- Downloads last month
- -