---

MarathonDocBench

A New Benchmark for Ultra-Long Document Parsing: Targeting Hundred-Page-Scale Documents.

MarathonDocBench is the first benchmark specifically designed for hundred-page document parsing. It contains 100 documents spanning 100–300 pages (~18k pages, ~205k annotated layout elements, mixed Chinese/English), with rich cross-page structures — cross-page text and tables appear frequently and densely across documents. Structural annotations (bounding boxes, block type, hierarchy, reading order, and cross-page continuations of text and tables) are initialized from a strong page-by-page parser and carefully refined by human annotators; content transcriptions are verified through an iterative VLM-based refinement pipeline, with flagged disagreements adjudicated by human annotators.

This repository ships the data (page images + ground-truth annotations). Evaluation reuses the MPDocBench-Parse toolkit — see Usage below.

---

1. Dataset

MarathonDocBench.json        # ground-truth annotations (100 documents)
images/
├── PDF1/page_1.png … 
├── PDF1/page_2.png …  
├── PDF1/ …                  # one folder of page images per document
├── PDF2/page_1.png
└── …                        # ~18k PNG pages total

Each entry in MarathonDocBench.json follows the MPDocBench schema (page_info / layout_dets / extra): layout_dets carries per-element category_type, text/html/latex, reading order, heading anno_id, and poly boxes; extra.relation lists cross-page truncated merges.

---

2. Usage

Evaluate with the MPDocBench-Parse toolkit; this dataset is a drop-in ground-truth replacement (same schema, same eight metrics).

# 1. Get the toolkit
git clone https://github.com/Tongyi-Zhiwen/Qwen-Doc.git
cd Qwen-Doc/MPDocBench
pip install -r requirements.txt
pip install mmeval          # needed by the dataset loader, not pinned upstream

# 1b. Replace 3 evaluator files with the accelerated drop-in from THIS repo
#     (metric-identical; only adds doc-level parallelism + a configurable match
#     timeout — vanilla hardcodes a 30s serial match that under-scores ultra-long
#     docs and can take days). Overwrites dataset/end2end_dataset.py,
#     metrics/cal_metric.py and utils/data_preprocess.py:
cp -r /path/to/MarathonDocBench/utils/mpdocbench_patch/* .

# 2. Place this dataset where the config can find it, e.g.
#    data/MarathonDocBench.json   and   data/images/...

# 3. Generate predictions: one Markdown file per document, named <doc_id>.md
#    (e.g. mp3lan_0101.md). Figure tags must follow the required format:
#        ![](pageN_x1_y1_x2_y2.jpg)
#    (N = 1-based page index; x1,y1,x2,y2 = integer coords on a 1000x1000 grid.)
#    Any parser is fine, as long as figure tags follow this format.

# 4. Point configs/end2end.yaml at the dataset (ground_truth.data_path ->
#    data/MarathonDocBench.json, prediction.data_path -> predictions/your_model_md),
#    then run with the settings used for the leaderboard. MPDOC_TEDS_TIMEOUT is
#    required: a few tables have tens of thousands of cells and pure-Python TEDS
#    would run for hours on them; past the cap such a table scores 0.
MPDOC_EVAL_NPROC=32 MPDOC_MATCH_TIMEOUT=1800 MPDOC_DOC_BUDGET=0 \
MPDOC_TEDS_NPROC=32 MPDOC_TEDS_TIMEOUT=300 \
    python pdf_validation.py --config ./configs/end2end.yaml

Per-metric results are written to result/. The leaderboard Overall is the mean of the eight metrics — compute it from the per-metric outputs.

• Running a general VLM. Whole-document inference can fail on provider caps: too many input images per request, request/JSON size limits, or the output-token ceiling (long docs truncate mid-table). Our competitor baselines send all pages in one request, progressively downscale images on oversize/timeout and retry, skip docs over the image cap, and leave failures as empty predictions so the denominator stays 100.
• Evaluation notes. Headings must be real Markdown headings (#, ##, …) — emphasis such as **bold** is NOT counted as a heading (it will lower HeadingTEDS). Figure page attribution is part of the score: a figure tagged with the wrong pageN is matched against the wrong page and penalized by FigureF1. The documents are very long; with the accelerated evaluator above (MPDOC_EVAL_NPROC + a large MPDOC_MATCH_TIMEOUT) a full 100-document run takes tens of minutes instead of hours.

---

3. Leaderboard

Sorted by Overall (the mean of the eight metrics), descending. All entries are end-to-end VLM-based parsers.

Model	Overall	TextEdit	T-TextEdit	TableTEDS	T-TableTEDS	FormulaCDM	ReadOrderEdit	FigureF1	HeadingTEDS
Unlimited OCR Works 2	77.06	89.6	87.7	74.8	71.1	87.9	85.3	71.8	48.2
Kimi-K3	33.7	52.5	33.6	27.8	26.3	40.5	38.9	21.4	28.9
Gemini-3.8-Flash	28.6	48.1	34.2	23.4	19.6	26.4	31.3	18.6	27.3
Qwen-3.8-Flash-Next	27.8	45.6	24.7	25.4	21.0	28.3	32.1	24.1	21.5
Gemini-3.1-Pro Preview	27.0	48.4	23.6	28.5	21.7	30.4	34.6	18.9	9.7
Qwen-3.8-Max	22.93	39.96	19.63	15.20	13.71	18.82	26.67	33.39	16.09
Minimax-M3	17.97	34.09	18.22	17.79	14.84	14.20	23.73	3.67	17.16
DeepSeek-4.1-Flash	12.34	26.00	6.67	12.74	12.59	10.45	15.70	6.95	7.62
Doubao-Seed-2.1-Turbo	8.50	18.38	8.97	5.87	5.07	6.00	10.18	8.16	5.37
GLM-5.3-Flash	7.51	14.01	6.43	7.12	6.65	4.50	7.70	7.10	6.55
Doubao-Seed-2.1-Pro	5.54	10.75	4.25	3.15	4.76	2.82	7.13	7.52	3.92

---

License & Copyright

Annotations and tooling references are released under Apache-2.0. The page images are derived from publicly accessible documents and are provided for academic and research use only; redistribution of the source documents is not granted by this license. If any content raises copyright concerns, contact the maintainers and it will be addressed promptly.

Acknowledgement

We build on MPDocBench-Parse for the evaluation toolkit and annotation schema, and thank OmniDocBench and CDM for the metrics and ideas this benchmark reuses.

Citation

@misc{shu2026unlimitedocrworks2,
      title={Unlimited OCR Works 2}, 
      author={Baidu ERNIE Team},
      year={2026},
      eprint={xx},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={xx}, 
}