Dataset Viewer
Auto-converted to Parquet Duplicate
benchmark
stringclasses
1 value
sessions_on_disk
int64
19.2k
19.2k
questions
int64
100
100
corpus_source
stringclasses
1 value
tool
stringclasses
7 values
version
stringclasses
7 values
reads_existing_history
bool
2 classes
agent_formats_supported
stringclasses
7 values
install_size
stringclasses
7 values
background_process
stringclasses
6 values
needs_model_or_key
bool
2 classes
mcp_schema_tokens_o200k
int64
0
8.28k
mcp_tool_count
int64
0
45
build_seconds
float64
13.6
10.8k
⌀
first_answer_ms
int64
26
4.5k
⌀
p50_search_ms
int64
97
2.6k
⌀
hit_at_1
int64
0
19
hit_at_5
int64
0
36
found_at_50
int64
0
65
notes
stringclasses
7 values
Day-Zero Cold Start Comparison
19,195
100
longmemeval_s_cleaned.json
deja-vu
0.23.2 (build) / 0.21.1 (MCP schema)
true
48
18.6 MB binary
none
false
477
1
13.6
26
97
19
36
65
Build time is mean of three cold builds on 0.23.2 (13.5-13.8 s); accuracy row build took 17.6 s. MCP schema collapsed to one tool with modes.
Day-Zero Cold Start Comparison
19,195
100
longmemeval_s_cleaned.json
ctx
1.4.12 (setup re-tested on 1.6.3, 2.2.8)
true
39 providers
43 MB binary
daemon
false
1,897
6
72
1,040
1,030
7
14
34
Explicit ctx import indexed 19,195 sessions (199,499 events, 553 MB store). Default ctx setup returned 0 sessions on a single project dir; on 2.2.8 across 200 dirs it completed in 72.7 s (hit@1 7, hit@5 18, found@50 37).
Day-Zero Cold Start Comparison
19,195
100
longmemeval_s_cleaned.json
CASS
0.8.0 (CLI) / main 38f0412 (accuracy)
true
25+
58.5 MB binary
none
false
0
0
3,360
380
370
5
8
16
CLI agent surface (no standing MCP schema overhead). Strict AND keyword semantics; evaluated with stop words stripped from queries.
Day-Zero Cold Start Comparison
19,195
100
longmemeval_s_cleaned.json
agentsview
0.44.0 (MCP) / 0.43.0 (accuracy)
true
62
115 MB binary
daemon, for search
false
8,283
7
236
1,440
113
7
13
19
Evaluated with --fts and stop words stripped. Synced 38,390 rows across both layouts (398,998 messages, 915 MB SQLite DB).
Day-Zero Cold Start Comparison
19,195
100
longmemeval_s_cleaned.json
agentmemory
0.9.29
true
Claude Code JSONL import (<=1000 files/call)
689 MB npm + 28 MB engine
worker + engine, four ports
false
745
7
95
185
145
14
34
65
Evaluated in documented keyless mode (BM25 only).
Day-Zero Cold Start Comparison
19,195
100
longmemeval_s_cleaned.json
MemPalace
3.9.0
true
file tree mining
312 MB + 168 MB embedding model
Chroma
true
8,183
45
10,800
4,500
2,600
14
20
46
Approximately 3 hours local mining via mempalace mine --mode convos over Claude layout.
Day-Zero Cold Start Comparison
19,195
100
longmemeval_s_cleaned.json
claude-mem
13.25.3
false
records forward from install
npm + Claude Code plugin
worker
true
2,307
15
null
null
null
0
0
0
Forward-only session-end hook recorder requiring an AI provider; holds no prior sessions on day zero.

Coding Agent Memory Benchmark

Machine-readable summary splits generated from the verified JSON artifacts in vshulcz/coding-agent-memory-benchmark (Zenodo archive: 10.5281/zenodo.24416712) for vshulcz/deja-vu.

Disclosure: published and maintained by the author of deja-vu. Upstream datasets carry their own licenses (LongMemEval: MIT; LoCoMo: CC-BY-NC-4.0). To respect LoCoMo's license, this dataset redistributes aggregate retrieval metrics and dataset digests only—no raw dialogue text or prompts are included.

Configs & Splits

Config Rows Contents Documentation
longmemeval_slices 14 Per-category and total retrieval metrics (hit@1/5/10/20, MRR, latency, sha256) on cleaned (470q) and full (500q) LongMemEval-S sets (deja-vu 0.23.2) Benchmarks guide
locomo_retrieval_metrics 6 Aggregate retrieval-only metrics over 1,982 LoCoMo questions (multi-hop, temporal, open-domain, single-hop, adversarial*, TOTAL) Benchmarks guide
day_zero_tools 7 Head-to-head cold-start build time, query latency, hit@1/5, found@50, and standing MCP tools/list schema overhead over 19,195 on-disk sessions Day-zero evaluation
task_cost_and_compaction 10 End-to-end repository task token/call ablations (11 runs per arm), 43-compaction retention measurements, and context-readiness baselines Day-zero evaluation

Reproducing & Submitting External Rankings

Clone the benchmark repository and score any external ranking JSON with Python 3 standard library only:

git clone https://github.com/vshulcz/coding-agent-memory-benchmark
cd coding-agent-memory-benchmark
python3 scripts/verify_artifacts.py
python3 scripts/score_longmemeval.py --data /path/to/longmemeval_s_cleaned.json --score submission.json --skip-abs
Downloads last month
-