| # Current CrossSearch grading protocol | |
| Version: gpt61-20261106. Scope: 4,650 existing answers (3,300 native-search and 1,350 uncapped search comparisons). | |
| Judge: GPT-6.1 Sol through the authorized inference gateway, high reasoning effort, temperature 0, 16,000 maximum response tokens. The semantic system prompt is [CrossSearch-GPT61-Judge-Prompt.txt](CrossSearch-GPT61-Judge-Prompt.txt). The judge received the original full task context, rubric, reference answer and candidate answer, without the generating model identity. No generator model was rerun. | |
| Each rubric ID must appear exactly once. Each item has a Boolean pass/fail decision, a rationale and verbatim evidence when supplied. The validator checks all quoted evidence against the unchanged answer and requires a normal completed response. Invalid or incomplete responses do not become grades; distinct completion attempts retain the same input and all previous observations are preserved. | |
| Weighted score = earned rubric weight divided by total rubric weight. Full credit requires score 1. Pass@1 is the mean of full-credit indicators across three attempts per task. Pass^3 is the share of tasks with all three attempts receiving full credit. Confidence intervals use 20,000 task-bootstrap resamples, keeping each task’s three attempts together. Search differences use paired resampling of the same 50 tasks. | |
| Original grades remain in the unchanged rollout shards and in run-index-original-20261105.jsonl.gz. Current grades are in run-index.jsonl.gz and grades-gpt61-20261106.jsonl.gz. Do not mix judge versions in a single reported metric. The independent human study used question-only context; these benchmark regrades additionally include full task context and reference answers. | |