ClawBench Lite: inspect the task schema before running an agent

Disclosure: this is an AI-assisted technical walkthrough of ClawBench, a project by Yuxuan Zhang and collaborators. It reports an offline inspection of public task definitions, not a new agent evaluation or reproduction of leaderboard scores.

A browser recording shows what an agent did. An intercepted request shows that a configured request rule was reached. Those are different pieces of evidence. Before starting a ClawBench experiment, check which evidence your chosen task and scoring workflow expect.

This guide gives you a small first step: inspect the 20-task Lite subset with standard Python, without setting up a model endpoint or browser container.

1. Pin the task definitions

The inspection below used commit d8faa73edd2241a367f8fc0d13320f8a376277c5. The files are in test-cases/v1-lite; Lite entries link back to their V1 task definitions.

For a fresh checkout:

git clone https://github.com/TIGER-AI-Lab/ClawBench.git
cd ClawBench
git checkout d8faa73edd2241a367f8fc0d13320f8a376277c5
cd ..

Use a fresh checkout to keep this inspection separate from any work in progress. Cloning needs network access; the Python inspection itself only reads local JSON files. No package installation is needed for the inspection.

2. Read the request-rule categories

Save the following as inspect_lite.py next to the checkout, then run python3 inspect_lite.py ClawBench.

"""Read task schemas from a ClawBench checkout; never execute browser tasks."""
import argparse
from collections import Counter
import json
from pathlib import Path

parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('checkout', type=Path, help='Path to a ClawBench source checkout')
args = parser.parse_args()
paths = sorted((args.checkout / 'test-cases/v1-lite').glob('*/task.json'))
if not paths:
    raise SystemExit('No Lite task definitions found; check the checkout path.')
counts = Counter()
cases = []
for path in paths:
    task = json.loads(path.read_text())
    schema = task.get('eval_schema', {})
    pattern = schema.get('url_pattern')
    if pattern == '__PLACEHOLDER_WILL_NOT_MATCH__':
        category = 'intentional_sentinel'
    elif isinstance(pattern, str) and pattern:
        category = 'other_nonempty_pattern'
    else:
        category = 'missing_or_empty_pattern'
    counts[category] += 1
    cases.append({'case': path.parent.name, 'category': category,
                  'method': schema.get('method')})
print(json.dumps({'files_read': len(paths), 'pattern_categories': dict(counts),
                  'cases': cases}, indent=2))

The script was run against the pinned checkout. Its summary was:

{
  "files_read": 20,
  "pattern_categories": {
    "intentional_sentinel": 8,
    "other_nonempty_pattern": 12
  }
}

The full output also lists each task name and configured HTTP method. The categories describe configuration at this revision; they are not pass rates, model results, or a validation of live endpoints.

3. Choose the evaluation workflow

In the native two-stage workflow, the request must first be intercepted, then the configured judge assesses whether its contents satisfy the instruction. --no-judge reports interception alone. The historical V1 workflow instead compares a recorded agent trajectory with a human reference under a separate rubric. See the pinned task walkthrough, evaluation guide, and scoring reference.

The distinction matters for interpreting intercepted: false: it cannot by itself tell you whether a useful trajectory exists, and enabling a request judge does not turn that trajectory into a judged request. Conversely, reaching a matching endpoint does not establish that the payload fulfills the user's instruction.

4. Carry provenance into the actual experiment

When you move beyond this file inspection, retain the task revision, suite, model, harness, judge/rubric, and denominator with the results. Inspect the recorded request and browser trajectory together rather than comparing unlabeled percentages.

Interception blocks requests that match the configured rule. Non-matching requests can still reach a live website; a task schema is not a blanket guarantee against side effects. This guide performs no browser actions. Actual agent runs require the environment and model configuration described in Getting started, and can incur model or infrastructure costs.

Continue with the released resources

• Paper
• Repository and current documentation
• Task dataset
• V1 trace dataset
• V2 trace dataset
• Leaderboard

A useful follow-up is one reproducible evaluation mismatch: the task revision, the expected criterion, and the evidence that led to a different verdict. Remove credentials and personal data from any trace you share. The goal is to make a score understandable enough that another researcher can check it.