Two Pure-Vision Pipelines at the Top of ViDoRe V3

Note: GPUs differ between entries, efficiency comparisons hold only under each entry’s evaluation environment.

TL;DR

• We built two retrieval pipelines for the ViDoRe V3 pipeline leaderboard. Both work on page images only. Both run open-weight models on a single GPU.
• VisAgentRet lets GLM-5.3-Flash use EVIE-4.5B as a search tool with its own queries and look at candidate pages. It reaches 70.01 average nDCG@10, the highest score on the pipeline leaderboard at the time of writing.
• EVIE-4.5B + JEV-27B-VL reranks the top 16 page images with one forward pass. It reaches 68.25 at 1.47 s per query and is listed third on the leaderboard.
• Code for both pipelines is on GitHub under Apache-2.0: visagentret and Jev-reranker.

The ViDoRe V3 pipeline leaderboard compares complete retrieval systems under one evaluation. A pipeline implements index() and search(), and the official CLI scores it on the 8 public English datasets of ViDoRe V3. When the leaderboard launched, the most accurate pipeline used text only (jina-embeddings-v4 + zerank-2), and the authors noted that visual rerankers still trailed the best text ones.

We wanted to see how far a pipeline can go when it only looks at visual pages. We submitted two pipelines at the two ends of the cost scale: an agent that searches and reads pages, and a reranker that looks at 16 pages once.

1. A shared first stage: EVIE-4.5B

Both pipelines start from Tencent's latest EVIE-4.5B, a late-interaction retriever over page images. It encodes each page as about 1,000 visual token vectors and scores a query against them with MaxSim. On its own, EVIE-4.5B scores above every single-model entry on the leaderboard. Its per-dataset scores are in Section 4.

2. VisAgentRet: an agent that looks at pages

GLM-5.3-Flash runs as a ReAct agent in the Codex CLI. We serve its NVFP4 checkpoint (nvidia/GLM-5.3-Flash-NVFP4) with vLLM on the same B300 GPU as EVIE-4.5B. It works through the two MCP tools shown above.

The prompt asks the agent to rewrite the question into focused queries, search several times, check promising pages and answer with a JSON list of at most 10 page ids. When the answer has fewer than 10 pages, we fill the rest with reciprocal rank fusion (k = 60) over all search results of the episode. An episode stops after 900 s, and the fusion fill then gives the whole ranking. This happened for 74 of 2,419 queries.

The agent only sees page images, retriever scores and document names (doc_id). In ViDoRe V3, a doc_id is often a descriptive title. Codex also offers a shell tool. The prompt tells the agent not to use it, and the agent ran 0 shell commands over all 2,419 episodes.

One episode analysis

Query from finance_en: "Morgan Stanley total Average Management VaR in 2024." EVIE-4.5B alone puts the only relevant page, 2239, in third place. The agent's first query is a slight rewrite, and EVIE-4.5B ranks 2239 second for it. The agent does this:

search  "Morgan Stanley total Average Management VaR 2024"              -> 2240, 2239, 2215, 53, 2237, ...
open    2240, 2239, 2215, 2237, 2258
search  "Total Management VaR Average $49 2024"                         -> 2239, 53, 2240, 1069, ...
search  "95% One-Day Management VaR Average Total 2024 Morgan Stanley"  -> 2240, 2239, 2237, 53, ...
open    2232
answer  {"ranking": [2239]}

The second query contains $49, a value the agent read from the pages it opened. The final answer holds page 2239 alone, and the fusion fill adds nine more pages after it. nDCG@10 for this query goes from 0.50 with EVIE-4.5B alone to 1.00.

Work per query

The NVIDIA rows come from their post, with tokens derived from their totals over 2,419 queries. Our agent searches more often than both NVIDIA agents and opens about 18 pages per query. It adds 3.25 points over its first stage, EVIE-4.5B alone.

3. EVIE-4.5B + JEV-27B-VL: one forward pass over 16 pages

JEV-27B-VL is a decision model: its decision head answers a typed question with one probability per option. We put the query and the top 16 EVIE-4.5B pages (896 px, labelled Page A to Page P with their document names) into the state and ask one multiple-choice question:

{
  "kind": "choice",
  "state": [
    "Query: <query>\nCandidate pages:",
    "\nPage A (document: <doc_id>):", {"image": "<page A>"},
    "...",
    "\nPage P (document: <doc_id>):", {"image": "<page P>"}
  ],
  "question": "Which page contains the information that best helps answer the query? Query: <query>",
  "options": ["Page A", "Page B", "...", "Page P"],
  "strategy": "single",
  "thinking": "off"
}

The server returns one probability per option from a single prefill of about 9.4k tokens. We sort the 16 pages by this probability and keep the EVIE-4.5B order for ranks 17 to 100. The 16 probabilities come from one softmax over the options, so they sum to one and measure each page against the other 15. This adds 1.49 points to EVIE-4.5B alone, with gains on 7 of 8 datasets.

The figure below shows one query with four relevant pages among the 16 candidates. EVIE-4.5B ranks the two grade-1 pages first. JEV-27B-VL moves the two grade-2 pages to the top and keeps page 2206 (grade 1) third with a probability of 0.22. nDCG@10 for this query goes from 0.76 to 0.92.

4. Per-dataset results and timing

nDCG@10 over 2,419 queries. Physics is the one dataset where EVIE-4.5B alone scores above both of our pipelines.

Search time per query

The other leaderboard entries were timed one query at a time, so we timed our pipelines the same way on the same GPUs (num_concurrent=1), on the first 30 hr-subset queries for the agent and the first 40 for the reranker. The reranker takes 1.47 s per query: 1.36 s for the JEV prefill and 0.04 s for the EVIE search. The agent takes 49.3 s per query.

hr was the slowest of the 8 datasets in our full agent run, so 49.3 s is likely a conservative estimate for the whole benchmark. The leaderboard lists 24.4 s and 1.05 s for our two entries because those full runs kept 16 and 8 queries in flight.

5. Page images or OCR text?

ViDoRe V3 ships a text version of every page, extracted with NV-Ingest, so we also ran the agent with that text in place of the page images. Everything else stays the same: the agent, the prompt structure, the EVIE-4.5B retriever over page images and the fusion fill. The only change is what the agent reads. search now returns the page text of each hit (up to 12,000 characters per page), and there is no get_page tool.

With OCR text, the agent scores 69.69 against 70.01 with page images, a difference of 0.32 points. Additionally, OCR text costs more context: 726k prompt tokens per query against 557k, with fewer LLM calls (12.1 against 17.7) and the same number of searches (14.3 against 14.2).

6. Reproduce

Each repository holds the exact pipeline class that the leaderboard links to, the server launchers and the job script that produced the results. The pipelines reach their model servers through environment variables listed in each README. For example, the reranker runs one dataset like this:

vidore-benchmark pipeline evaluate --dataset-name vidore/vidore_v3_hr --language english \
    --module-path src/jevrank_pipeline.py --class-name JevRerankPipeline \
    --pipeline-args '{"retriever_model": "tencent/EVIE-4.5B", "reranker_model": "autotrust/JEV-27B-VL", "candidate_k": 16, "retriever_top_k": 100, "page_max_side": 896, "page_jpeg_quality": 85, "num_concurrent": 8, "max_retries": 2}'

Both pipelines were evaluated with vidore-benchmark a70f23a on the 8 English datasets.

Links

• ViDoRe V3 pipeline leaderboard, the ViDoRe V3 paper and the pipeline leaderboard post
• EVIE: tencent/EVIE-4.5B and the paper EVIE: Evidence-Vector-Informed Embeddings for Visual Document Retrieval
• NVIDIA: NeMo Retriever's agentic retrieval pipeline
• Our JEV reranker paper: Multimodal Reranking Is a System One Decision
• JEV: autotrust/JEV-27B-VL and its base model Qwen/Qwen3.8-27B
• GLM-5.3-Flash: zai-org/GLM-5.3-Flash and the NVFP4 checkpoint we serve, nvidia/GLM-5.3-Flash-NVFP4
• More findings on Jev-style model as reranker: Multimodal Reranking Is a System One Decision (project page,blog)

Acknowledgements

The computations were performed using computer resources provided by the Aalto University School of Science “Science-IT” project.