The dataset viewer is not available for this split.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
schema: string
taxonomy: string
runId: string
taskKey: string
dag_sha256: string
outcome: struct<success: int64, reward: double, source: string, raw: struct<success: int64, all_success: int6 (... 902 chars omitted)
child 0, success: int64
child 1, reward: double
child 2, source: string
child 3, raw: struct<success: int64, all_success: int64, deterministic: int64, state_identical: int64, determinism (... 841 chars omitted)
child 0, success: int64
child 1, all_success: int64
child 2, deterministic: int64
child 3, state_identical: int64
child 4, determinism_checked: int64
child 5, instance_ok: int64
child 6, n_actions: int64
child 7, replays: int64
child 8, sim_steps: int64
child 9, sim_time_s: double
child 10, timed_out: int64
child 11, replay_wall_s: double
child 12, replay_timeout_s: double
child 13, grader_error: int64
child 14, infra_error: int64
child 15, replay_retries: int64
child 16, replay_gpu_fault: int64
child 17, agent_sessions: int64
child 18, agent_resets: int64
child 19, agent_sim_steps: int64
child 20, agent_records: int64
child 21, agent_replays: int64
child 22, agent_cost_self_reported: int64
child 23, video_rendered: int64
child 24, video_episode_success: int64
child 25, resource_enforced: int64
child 26, limits_violation_observed: int64
child 27, resource_monitor_missing: int64
c
...
hild 9, step: struct<cat: string, item: string>
child 0, cat: string
child 1, item: string
child 10, contact: list<item: struct<who: string, how: string, note_zh: string, plan: int64>>
child 0, item: struct<who: string, how: string, note_zh: string, plan: int64>
child 0, who: string
child 1, how: string
child 2, note_zh: string
child 3, plan: int64
child 11, optional: bool
child 12, verify_zh: null
child 13, sources: list<item: string>
child 0, item: string
child 14, conf: int64
child 15, note_zh: string
child 6, constraints: list<item: null>
child 0, item: null
child 7, success: null
child 8, atoms: list<item: struct<atom: string, level: string, source: string>>
child 0, item: struct<atom: string, level: string, source: string>
child 0, atom: string
child 1, level: string
child 2, source: string
task_class: string
suite: string
evidence_check: struct<n_state_runs: int64, n_faithful: int64, success_runs_reaching_all: string, spec_success_eq_gr (... 76 chars omitted)
child 0, n_state_runs: int64
child 1, n_faithful: int64
child 2, success_runs_reaching_all: string
child 3, spec_success_eq_grader: string
child 4, diverged_runs: list<item: null>
child 0, item: null
child 5, notes_zh: list<item: string>
child 0, item: string
to
{'schema': Value('string'), 'taxonomy': Value('string'), 'taskKey': Value('string'), 'suite': Value('string'), 'base': Value('string'), 'title_zh': Value('string'), 'summary_zh': Value('string'), 'embodiment_zh': Value('string'), 'task_class': Value('string'), 'analysis_level': Value('string'), 'spec': {'version': Value('string'), 'task': Value('string'), 'bench': Value('string'), 'progress': Value('string'), 'entities': {}, 'milestones': List({'id': Value('string'), 'zh': Value('string'), 'mode': Value('string'), 'when': Value('null'), 'after': List(Value('string')), 'hold_s': Value('int64'), 'freeze_held': Value('bool'), 'goal': Value('bool'), 'criterion_zh': Value('string'), 'step': {'cat': Value('string'), 'item': Value('string')}, 'contact': List({'who': Value('string'), 'how': Value('string'), 'note_zh': Value('string'), 'plan': Value('int64')}), 'optional': Value('bool'), 'verify_zh': Value('null'), 'sources': List(Value('string')), 'conf': Value('int64'), 'note_zh': Value('string')}), 'constraints': List(Value('null')), 'success': Value('null'), 'atoms': List({'atom': Value('string'), 'level': Value('string'), 'source': Value('string')})}, 'native_metrics': List({'name': Value('string'), 'unit': Value('string'), 'threshold': Value('float64'), 'direction': Value('string'), 'note_zh': Value('string')}), 'open_questions_zh': List(Value('string')), 'conf': Value('int64'), 'conf_reason_zh': Value('string'), 'review': {'status': Value('string'), 'rounds': Value('int64'), 'reviewers': List({'engine': Value('string'), 'model': Value('string'), 'round': Value('int64'), 'verdict': Value('string'), 'conf': Value('int64'), 'summary_zh': Value('string'), 'n_issues': Value('int64'), 'n_major': Value('int64')}), 'open_zh': List(Value('null'))}, 'evidence_check': {'n_state_runs': Value('int64'), 'n_faithful': Value('int64'), 'success_runs_reaching_all': Value('string'), 'spec_success_eq_grader': Value('string'), 'diverged_runs': List(Value('null')), 'notes_zh': List(Value('string'))}, 'provenance': {'prompt': Value('string'), 'prompt_sha256': Value('string'), 'engine': Value('string'), 'model': Value('string'), 'effort': Value('string'), 'created': Value('timestamp[s]'), 'review_prompt': Value('string'), 'review_prompt_sha256': Value('string'), 'reconcile_prompt': Value('string'), 'reconcile_prompt_sha256': Value('string'), 'calls': List({'stage': Value('string'), 'round': Value('int64'), 'ok': Value('bool'), 'space': Value('string')}), 'review_engine': Value('string'), 'review_model': Value('string'), 'inputs_manifest_sha256': Value('string'), 'repo_commit': Value('string')}}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 149, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 129, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 544, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 3052, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2559, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2584, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 558, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 440, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 361, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2381, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2309, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
schema: string
taxonomy: string
runId: string
taskKey: string
dag_sha256: string
outcome: struct<success: int64, reward: double, source: string, raw: struct<success: int64, all_success: int6 (... 902 chars omitted)
child 0, success: int64
child 1, reward: double
child 2, source: string
child 3, raw: struct<success: int64, all_success: int64, deterministic: int64, state_identical: int64, determinism (... 841 chars omitted)
child 0, success: int64
child 1, all_success: int64
child 2, deterministic: int64
child 3, state_identical: int64
child 4, determinism_checked: int64
child 5, instance_ok: int64
child 6, n_actions: int64
child 7, replays: int64
child 8, sim_steps: int64
child 9, sim_time_s: double
child 10, timed_out: int64
child 11, replay_wall_s: double
child 12, replay_timeout_s: double
child 13, grader_error: int64
child 14, infra_error: int64
child 15, replay_retries: int64
child 16, replay_gpu_fault: int64
child 17, agent_sessions: int64
child 18, agent_resets: int64
child 19, agent_sim_steps: int64
child 20, agent_records: int64
child 21, agent_replays: int64
child 22, agent_cost_self_reported: int64
child 23, video_rendered: int64
child 24, video_episode_success: int64
child 25, resource_enforced: int64
child 26, limits_violation_observed: int64
child 27, resource_monitor_missing: int64
c
...
hild 9, step: struct<cat: string, item: string>
child 0, cat: string
child 1, item: string
child 10, contact: list<item: struct<who: string, how: string, note_zh: string, plan: int64>>
child 0, item: struct<who: string, how: string, note_zh: string, plan: int64>
child 0, who: string
child 1, how: string
child 2, note_zh: string
child 3, plan: int64
child 11, optional: bool
child 12, verify_zh: null
child 13, sources: list<item: string>
child 0, item: string
child 14, conf: int64
child 15, note_zh: string
child 6, constraints: list<item: null>
child 0, item: null
child 7, success: null
child 8, atoms: list<item: struct<atom: string, level: string, source: string>>
child 0, item: struct<atom: string, level: string, source: string>
child 0, atom: string
child 1, level: string
child 2, source: string
task_class: string
suite: string
evidence_check: struct<n_state_runs: int64, n_faithful: int64, success_runs_reaching_all: string, spec_success_eq_gr (... 76 chars omitted)
child 0, n_state_runs: int64
child 1, n_faithful: int64
child 2, success_runs_reaching_all: string
child 3, spec_success_eq_grader: string
child 4, diverged_runs: list<item: null>
child 0, item: null
child 5, notes_zh: list<item: string>
child 0, item: string
to
{'schema': Value('string'), 'taxonomy': Value('string'), 'taskKey': Value('string'), 'suite': Value('string'), 'base': Value('string'), 'title_zh': Value('string'), 'summary_zh': Value('string'), 'embodiment_zh': Value('string'), 'task_class': Value('string'), 'analysis_level': Value('string'), 'spec': {'version': Value('string'), 'task': Value('string'), 'bench': Value('string'), 'progress': Value('string'), 'entities': {}, 'milestones': List({'id': Value('string'), 'zh': Value('string'), 'mode': Value('string'), 'when': Value('null'), 'after': List(Value('string')), 'hold_s': Value('int64'), 'freeze_held': Value('bool'), 'goal': Value('bool'), 'criterion_zh': Value('string'), 'step': {'cat': Value('string'), 'item': Value('string')}, 'contact': List({'who': Value('string'), 'how': Value('string'), 'note_zh': Value('string'), 'plan': Value('int64')}), 'optional': Value('bool'), 'verify_zh': Value('null'), 'sources': List(Value('string')), 'conf': Value('int64'), 'note_zh': Value('string')}), 'constraints': List(Value('null')), 'success': Value('null'), 'atoms': List({'atom': Value('string'), 'level': Value('string'), 'source': Value('string')})}, 'native_metrics': List({'name': Value('string'), 'unit': Value('string'), 'threshold': Value('float64'), 'direction': Value('string'), 'note_zh': Value('string')}), 'open_questions_zh': List(Value('string')), 'conf': Value('int64'), 'conf_reason_zh': Value('string'), 'review': {'status': Value('string'), 'rounds': Value('int64'), 'reviewers': List({'engine': Value('string'), 'model': Value('string'), 'round': Value('int64'), 'verdict': Value('string'), 'conf': Value('int64'), 'summary_zh': Value('string'), 'n_issues': Value('int64'), 'n_major': Value('int64')}), 'open_zh': List(Value('null'))}, 'evidence_check': {'n_state_runs': Value('int64'), 'n_faithful': Value('int64'), 'success_runs_reaching_all': Value('string'), 'spec_success_eq_grader': Value('string'), 'diverged_runs': List(Value('null')), 'notes_zh': List(Value('string'))}, 'provenance': {'prompt': Value('string'), 'prompt_sha256': Value('string'), 'engine': Value('string'), 'model': Value('string'), 'effort': Value('string'), 'created': Value('timestamp[s]'), 'review_prompt': Value('string'), 'review_prompt_sha256': Value('string'), 'reconcile_prompt': Value('string'), 'reconcile_prompt_sha256': Value('string'), 'calls': List({'stage': Value('string'), 'round': Value('int64'), 'ok': Value('bool'), 'space': Value('string')}), 'review_engine': Value('string'), 'review_model': Value('string'), 'inputs_manifest_sha256': Value('string'), 'repo_commit': Value('string')}}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
EFE failure analysis labels
Failure-analysis labels for runs of the Embodied First Exam (EFE) benchmark. Per task: a task DAG (semantic steps, step types, contact, prerequisite edges). Per run (trajectory): when each step holds on the run's video (progress map), the contact that actually happened, and for a failed run the breakpoint step, the failure phenomena and the root causes. Label text is in Chinese.
The labels come in two phases: phase 1 (folder phase1/, prompts v3.1, sections below) and phase 2 (folder phase2/,
prompts v3.2, section Phase 2 at the end). For a task in both folders, use phase 2: the importer
reads only the highest phaseN/ folder of a task (its DAG with its own trajectories).
- Prompts: v3.1 (frozen labelling prompts, taxonomy v2; schemas
efe-task-dag/1andefe-trajectory-labels/1). - Writer: Codex
gpt-6.1-sol(effort high for task DAGs and trajectories). Reviewer: Claudeclaude-sonnet-5-5; up to two fix rounds plus a confirm round; files still disputed after that carryreview.status: "disputed". - Task repository: embodied-first-exam commit
4c6d01b9990e3b78ddb620bbd237932b3696f60e. - Phase 1: two runs per task (one success when there is one, plus a failure of another model; runs with a video first).
Run records and videos: dataset
MLL-Lab/embodied-exam-data,runs/<runId>. - Phase 1 complete: every task of the run passed validation and is listed below. Status: 331 tasks (3 pilot, 328 full run), 662 trajectories.
Layout
phase1/<suite>/<task>/task_dag.json: the task DAG;phase1/<suite>/<task>/trajectories/<runId>.json: one label file per run. In these copies the provenance work-directory paths (calls[].space) are relative anddag_sha256is the sha256 oftask_dag.jsonas published here.- Pilot tasks (marked
pilotbelow) were labelled with the same prompts, models and efforts in a 3-task pilot before the full run.pilot_xhigh_dags/holds the same 3 task DAGs written at effort xhigh, for comparison only (no trajectories were labelled against them).
Tasks
| folder | trajectories | source |
|---|---|---|
phase1/ambench/cabinet_pick_place/ |
2 | full run |
phase1/behavior-1k/assembling_gift_baskets/ |
2 | full run |
phase1/behavior-1k/boxing_books_up_for_storage/ |
2 | full run |
phase1/behavior-1k/boxing_food_after_dinner/ |
2 | full run |
phase1/behavior-1k/bringing_in_wood/ |
2 | full run |
phase1/behavior-1k/bringing_water/ |
2 | full run |
phase1/behavior-1k/can_meat/ |
2 | full run |
phase1/behavior-1k/canning_food/ |
2 | full run |
phase1/behavior-1k/carrying_in_groceries/ |
2 | full run |
phase1/behavior-1k/chopping_wood/ |
2 | full run |
phase1/behavior-1k/clean_a_patio/ |
2 | full run |
phase1/behavior-1k/clean_a_trumpet/ |
2 | full run |
phase1/behavior-1k/clean_up_your_desk/ |
2 | full run |
phase1/behavior-1k/clean_your_rusty_garden_tools/ |
2 | full run |
phase1/behavior-1k/cleaning_up_branches_and_twigs/ |
2 | full run |
phase1/behavior-1k/cleaning_up_plates_and_food/ |
2 | full run |
phase1/behavior-1k/clearing_food_from_table_into_fridge/ |
2 | full run |
phase1/behavior-1k/collecting_aluminum_cans/ |
2 | full run |
phase1/behavior-1k/collecting_childrens_toys/ |
2 | full run |
phase1/behavior-1k/cook_a_brisket/ |
2 | full run |
phase1/behavior-1k/cook_a_frozen_pie/ |
2 | full run |
phase1/behavior-1k/cook_bacon/ |
2 | pilot |
phase1/behavior-1k/cook_broccolini/ |
2 | full run |
phase1/behavior-1k/cook_brussels_sprouts/ |
2 | full run |
phase1/behavior-1k/cook_cabbage/ |
2 | full run |
phase1/behavior-1k/cook_hot_dogs/ |
2 | full run |
phase1/behavior-1k/dispose_of_batteries/ |
2 | full run |
phase1/behavior-1k/freeze_fruit/ |
2 | full run |
phase1/behavior-1k/freeze_pies/ |
2 | full run |
phase1/behavior-1k/laying_tile_floors/ |
2 | full run |
phase1/behavior-1k/loading_the_car/ |
2 | full run |
phase1/behavior-1k/make_gift_bags_for_baby_showers/ |
2 | full run |
phase1/behavior-1k/make_pizza/ |
2 | full run |
phase1/behavior-1k/make_rose_centerpieces/ |
2 | full run |
phase1/behavior-1k/organizing_art_supplies/ |
2 | full run |
phase1/behavior-1k/organizing_school_stuff/ |
2 | full run |
phase1/behavior-1k/outfit_a_basic_toolbox/ |
2 | full run |
phase1/behavior-1k/packing_meal_for_delivery/ |
2 | full run |
phase1/behavior-1k/preparing_lunch_box/ |
2 | full run |
phase1/behavior-1k/put_together_a_basic_pruning_kit/ |
2 | full run |
phase1/behavior-1k/putting_away_toys/ |
2 | full run |
phase1/behavior-1k/putting_dirty_dishes_in_sink/ |
2 | full run |
phase1/behavior-1k/putting_dishes_away_after_cleaning/ |
2 | full run |
phase1/behavior-1k/putting_up_Christmas_decorations_inside/ |
2 | full run |
phase1/behavior-1k/re_shelving_library_books/ |
2 | full run |
phase1/behavior-1k/rearrange_your_room/ |
2 | full run |
phase1/behavior-1k/rearranging_kitchen_furniture/ |
2 | full run |
phase1/behavior-1k/setting_the_fire/ |
2 | full run |
phase1/behavior-1k/setting_the_table/ |
2 | full run |
phase1/behavior-1k/setup_a_bar_for_a_cocktail_party/ |
2 | full run |
phase1/behavior-1k/slicing_vegetables/ |
2 | full run |
phase1/behavior-1k/sorting_books_on_shelf/ |
2 | full run |
phase1/behavior-1k/sorting_bottles_cans_and_paper/ |
2 | full run |
phase1/behavior-1k/sorting_household_items/ |
2 | full run |
phase1/behavior-1k/sorting_vegetables/ |
2 | full run |
phase1/behavior-1k/storing_food/ |
2 | full run |
phase1/behavior-1k/sweeping_garage/ |
2 | full run |
phase1/behavior-1k/tidying_living_room/ |
2 | full run |
phase1/behavior-1k/unloading_the_car/ |
2 | full run |
phase1/behavior-1k/vacuuming_floors/ |
2 | full run |
phase1/behavior-1k/wash_a_baseball_cap/ |
2 | full run |
phase1/behavior-1k/wash_dog_toys/ |
2 | full run |
phase1/bigym2/dishwasher_close/ |
2 | full run |
phase1/bigym2/dishwasher_load_plates/ |
2 | full run |
phase1/bigym2/flip_cutlery/ |
2 | full run |
phase1/bigym2/move_two_plates/ |
2 | full run |
phase1/bigym2/pick_box/ |
2 | full run |
phase1/bigym2/put_cups/ |
2 | full run |
phase1/bigym2/sandwich_remove/ |
2 | full run |
phase1/bigym2/saucepan_to_hob/ |
2 | full run |
phase1/bigym2/wall_cupboard_open/ |
2 | full run |
phase1/dexhand/assemble_hamburger/ |
2 | full run |
phase1/dexhand/batter_whisk/ |
2 | full run |
phase1/dexhand/bicycle_pump_stroke/ |
2 | full run |
phase1/dexhand/binder_page_load/ |
2 | full run |
phase1/dexhand/boot_sole_brush/ |
2 | full run |
phase1/dexhand/drying_rack_unfold/ |
2 | full run |
phase1/dexhand/envelope_letter_insert/ |
2 | full run |
phase1/dexhand/essential_pen_regrasp/ |
2 | full run |
phase1/dexhand/essential_rotary_gait/ |
2 | full run |
phase1/dexhand/fan_guard_install/ |
2 | full run |
phase1/dexhand/picture_hang/ |
2 | full run |
phase1/dexhand/stool_brace_insert/ |
2 | full run |
phase1/dexhand/sweep_dustpan/ |
2 | full run |
phase1/dexhand/usb_plug_insert/ |
2 | full run |
phase1/dexhand/vacuum_bin_reseat/ |
2 | full run |
phase1/dexjoco/bimanual_photograph/ |
2 | full run |
phase1/dextoolbench/brush_blue_brush_sweep_forward/ |
2 | full run |
phase1/dextoolbench/brush_red_brush_sweep_forward/ |
2 | full run |
phase1/dextoolbench/brush_red_brush_sweep_right/ |
2 | full run |
phase1/dextoolbench/eraser_handle_eraser_wipe_c/ |
2 | full run |
phase1/dextoolbench/hammer_claw_hammer_swing_down/ |
2 | full run |
phase1/dextoolbench/hammer_claw_hammer_swing_side/ |
2 | full run |
phase1/dextoolbench/hammer_mallet_hammer_swing_side/ |
2 | full run |
phase1/dextoolbench/marker_sharpie_marker_draw_smile/ |
2 | full run |
phase1/dextoolbench/marker_sharpie_marker_write_c/ |
2 | full run |
phase1/dextoolbench/marker_staples_marker_draw_smile/ |
2 | full run |
phase1/dextoolbench/marker_staples_marker_write_c/ |
2 | full run |
phase1/dextoolbench/screwdriver_long_screwdriver_spin_horizontal/ |
2 | full run |
phase1/dextoolbench/screwdriver_long_screwdriver_spin_vertical/ |
2 | full run |
phase1/dextoolbench/screwdriver_short_screwdriver_spin_vertical/ |
2 | full run |
phase1/dextoolbench/spatula_flat_spatula_flip_over/ |
2 | full run |
phase1/dextoolbench/spatula_flat_spatula_serve_plate/ |
2 | full run |
phase1/dextoolbench/spatula_spoon_spatula_flip_over/ |
2 | full run |
phase1/dextoolbench/spatula_spoon_spatula_serve_plate/ |
2 | full run |
phase1/embodied-olympics-fun/badminton_serve/ |
2 | full run |
phase1/embodied-olympics-fun/balance_board/ |
2 | full run |
phase1/embodied-olympics-fun/box_jump/ |
2 | full run |
phase1/embodied-olympics-fun/clean_and_jerk/ |
2 | full run |
phase1/embodied-olympics-fun/dunk_between_legs/ |
2 | full run |
phase1/embodied-olympics-fun/high_jump/ |
2 | full run |
phase1/embodied-olympics-fun/hurdles_freestanding/ |
2 | full run |
phase1/embodied-olympics-fun/juggling_head/ |
2 | full run |
phase1/embodied-olympics-fun/long_rope/ |
2 | full run |
phase1/embodied-olympics-fun/pole_vault/ |
2 | full run |
phase1/embodied-olympics-fun/standing_long_jump/ |
2 | full run |
phase1/embodied-olympics-fun/steeplechase/ |
2 | full run |
phase1/embodied-olympics-fun/triple_jump/ |
2 | full run |
phase1/embodied-olympics-fun/vault/ |
2 | full run |
phase1/embodied-olympics-fun/volleyball_spike/ |
2 | pilot |
phase1/embodied-olympics-match/badminton_duelrl_i01/ |
2 | full run |
phase1/embodied-olympics-match/badminton_match_i01/ |
2 | full run |
phase1/embodied-olympics-match/badminton_match_i02/ |
2 | full run |
phase1/embodied-olympics-match/badminton_match_i04/ |
2 | full run |
phase1/embodied-olympics-match/badminton_matchlow_i01/ |
2 | full run |
phase1/embodied-olympics-match/fencing_bout_i01/ |
2 | full run |
phase1/embodied-olympics-match/foosball_match_i01/ |
2 | full run |
phase1/embodied-olympics-match/soccer_jointgpu_i01/ |
2 | full run |
phase1/embodied-olympics-match/soccer_match/ |
2 | full run |
phase1/embodied-olympics-match/soccerssl_match/ |
2 | full run |
phase1/embodied-olympics-match/soccerssl_match_i01/ |
2 | full run |
phase1/embodied-olympics-match/tabletennis_duelrl_i01/ |
2 | full run |
phase1/embodied-olympics-match/tabletennis_match_i01/ |
2 | full run |
phase1/embodied-olympics-match/tabletennis_match_i02/ |
2 | full run |
phase1/embodied-olympics-match/tabletennis_match_i03/ |
2 | full run |
phase1/embodied-olympics-match/tabletennis_matchlow_i01/ |
2 | full run |
phase1/embodied-safety-exam/a01_i01/ |
2 | full run |
phase1/embodied-safety-exam/a01_i02/ |
2 | full run |
phase1/embodied-safety-exam/a01_i08/ |
2 | full run |
phase1/embodied-safety-exam/a02_i03/ |
2 | full run |
phase1/embodied-safety-exam/a04/ |
2 | full run |
phase1/embodied-safety-exam/a06_i01/ |
2 | full run |
phase1/embodied-safety-exam/a06_i03/ |
2 | full run |
phase1/embodied-safety-exam/a08_i01/ |
2 | full run |
phase1/embodied-safety-exam/a17/ |
2 | full run |
phase1/embodied-safety-exam/a26_i02/ |
2 | full run |
phase1/embodied-safety-exam/a27_i06/ |
2 | full run |
phase1/embodied-safety-exam/a28/ |
2 | full run |
phase1/embodied-safety-exam/a28_i01/ |
2 | full run |
phase1/embodied-safety-exam/a28_i03/ |
2 | full run |
phase1/embodied-safety-exam/a29_i01/ |
2 | full run |
phase1/embodied-safety-exam/a29_i03/ |
2 | full run |
phase1/embodied-safety-exam/a33_i01/ |
2 | full run |
phase1/embodied-safety-exam/a33_i04/ |
2 | full run |
phase1/embodied-safety-exam/a34_i07/ |
2 | full run |
phase1/embodied-safety-exam/a34_i09/ |
2 | full run |
phase1/embodied-safety-exam/a34_i11/ |
2 | full run |
phase1/embodied-safety-exam/b06_i01/ |
2 | full run |
phase1/embodied-safety-exam/b06_i02/ |
2 | full run |
phase1/embodied-safety-exam/b11/ |
2 | full run |
phase1/embodied-safety-exam/b11_i01/ |
2 | full run |
phase1/embodied-safety-exam/b11_i03/ |
2 | full run |
phase1/embodied-safety-exam/b12_i04/ |
2 | full run |
phase1/embodied-safety-exam/b12_i06/ |
2 | full run |
phase1/embodied-safety-exam/b12_i07/ |
2 | full run |
phase1/embodied-safety-exam/c02/ |
2 | full run |
phase1/embodied-safety-exam/c05_i01/ |
2 | full run |
phase1/embodied-safety-exam/c05_i02/ |
2 | full run |
phase1/embodied-safety-exam/c16/ |
2 | full run |
phase1/embodied-safety-exam/d12_i04/ |
2 | full run |
phase1/embodied-safety-exam/e06/ |
2 | full run |
phase1/embodied-safety-exam/e06_i03/ |
2 | full run |
phase1/embodied-safety-exam/f01/ |
2 | full run |
phase1/embodied-safety-exam/f04_i01/ |
2 | full run |
phase1/embodied-safety-exam/f04_i02/ |
2 | full run |
phase1/embodied-safety-exam/f05/ |
2 | full run |
phase1/embodied-safety-exam/f05_i01/ |
2 | full run |
phase1/embodied-safety-exam/f06/ |
2 | full run |
phase1/embodied-safety-exam/f06_i01/ |
2 | full run |
phase1/embodied-safety-exam/f08/ |
2 | full run |
phase1/embodied-safety-exam/f09/ |
2 | full run |
phase1/embodied-safety-exam/f11/ |
2 | full run |
phase1/humanoidbench/loco_balance_hard/ |
2 | full run |
phase1/humanoidbench/loco_hurdle/ |
2 | full run |
phase1/humanoidbench/manip_bookshelf_simple/ |
2 | full run |
phase1/humanoidbench/manip_door/ |
2 | full run |
phase1/humanoidbench/manip_powerlift/ |
2 | full run |
phase1/humanoidbench/manip_room/ |
2 | full run |
phase1/humanoidbench/manip_window/ |
2 | full run |
phase1/ispatialgym/alignment_axial/ |
2 | full run |
phase1/ispatialgym/camera_relocalization_obstructed/ |
2 | full run |
phase1/ispatialgym/configuration_matching_articulation_open_small_scale/ |
2 | full run |
phase1/ispatialgym/configuration_matching_block_design/ |
2 | full run |
phase1/ispatialgym/inter_object_relation_satisfaction_nextto_small_scale/ |
2 | full run |
phase1/ispatialgym/inter_object_relation_satisfaction_ontop_small_scale/ |
2 | full run |
phase1/ispatialgym/inter_object_relation_satisfaction_touching_small_scale/ |
2 | full run |
phase1/kinder/scoop_pour_i01/ |
2 | full run |
phase1/kinder/sweep_into_drawer/ |
2 | full run |
phase1/kinder/sweep_simple_i01/ |
2 | full run |
phase1/libero-max/goal_wine_bottle_on_wine_rack_top_visual_theme_switch/ |
2 | full run |
phase1/maniguard/cabinet_pickup_i22/ |
2 | full run |
phase1/maniguard/cabinet_pickup_i26/ |
2 | full run |
phase1/maniguard/dusty_transfer_i03/ |
2 | full run |
phase1/maniguard/dusty_transfer_i16/ |
2 | full run |
phase1/maniguard/jar_transport_i13/ |
2 | full run |
phase1/maniguard/lid_transport_i12/ |
2 | full run |
phase1/metaworldplus/button_press_topdown/ |
2 | full run |
phase1/molmospaces/pick/ |
2 | full run |
phase1/mujoco-playground/aloha_single_peg_insertion/ |
2 | full run |
phase1/mujoco-playground/barkour_joystick/ |
2 | full run |
phase1/mujoco-playground/go1_handstand/ |
2 | full run |
phase1/mujoco-playground/go1_joystick_flat_terrain/ |
2 | full run |
phase1/mujoco-playground/spot_flat_terrain_joystick/ |
2 | full run |
phase1/mujoco-playground/spot_joystick_gait_tracking/ |
2 | full run |
phase1/myosuite/baoding/ |
2 | full run |
phase1/myosuite/bimanual/ |
2 | full run |
phase1/myosuite/chase_tag/ |
2 | full run |
phase1/myosuite/dex_airplane_fly/ |
2 | full run |
phase1/myosuite/dex_apple_lift/ |
2 | full run |
phase1/myosuite/dex_bunny_inspect/ |
2 | full run |
phase1/myosuite/dex_cup_pour/ |
2 | full run |
phase1/myosuite/dex_cylinder_pass/ |
2 | full run |
phase1/myosuite/die_reorient/ |
2 | pilot |
phase1/myosuite/obj_reorient/ |
2 | full run |
phase1/myosuite/osl_run/ |
2 | full run |
phase1/myosuite/soccer/ |
2 | full run |
phase1/pandadyn/d01_reciprocal_cup_relay/ |
2 | full run |
phase1/pandadyn/d03_cooperative_puck_course/ |
2 | full run |
phase1/pandadyn/d07_spring_gate_switchyard/ |
2 | full run |
phase1/pandadyn/d09_lever_launch_catch_reload/ |
2 | full run |
phase1/pandadyn/s01_cup_route_delivery/ |
2 | full run |
phase1/pandadyn/s06_cube_flip_route_recover/ |
2 | full run |
phase1/pandadyn/s09_toss_recatch_delivery/ |
2 | full run |
phase1/pandadyn/s11_cylinder_rescue_route_deliver/ |
2 | full run |
phase1/pandadyn/s12_moving_bridge_puck_relay/ |
2 | full run |
phase1/roboarc/blanket_flatten_02/ |
2 | full run |
phase1/roboarc/blanket_flatten_03/ |
2 | full run |
phase1/roboarc/blanket_flatten_07/ |
2 | full run |
phase1/roboarc/blanket_flatten_08/ |
2 | full run |
phase1/roboarc/blanket_fold_01/ |
2 | full run |
phase1/roboarc/blanket_fold_03/ |
2 | full run |
phase1/roboarc/blanket_fold_04/ |
2 | full run |
phase1/roboarc/blanket_fold_05/ |
2 | full run |
phase1/roboarc/buzz_wire_3d_01/ |
2 | full run |
phase1/roboarc/buzz_wire_3d_02/ |
2 | full run |
phase1/roboarc/buzz_wire_3d_04/ |
2 | full run |
phase1/roboarc/buzz_wire_3d_06/ |
2 | full run |
phase1/roboarc/buzz_wire_3d_07/ |
2 | full run |
phase1/roboarc/buzz_wire_3d_09/ |
2 | full run |
phase1/roboarc/buzz_wire_3d_10/ |
2 | full run |
phase1/roboarc/cast_coil_04/ |
2 | full run |
phase1/roboarc/cast_coil_06/ |
2 | full run |
phase1/roboarc/cast_coil_08/ |
2 | full run |
phase1/roboarc/cast_keyhole_03/ |
2 | full run |
phase1/roboarc/cast_keyhole_04/ |
2 | full run |
phase1/roboarc/cast_keyhole_05/ |
2 | full run |
phase1/roboarc/fiber_bundling_04/ |
2 | full run |
phase1/roboarc/fiber_bundling_06/ |
2 | full run |
phase1/roboarc/fiber_bundling_07/ |
2 | full run |
phase1/roboarc/fiber_bundling_09/ |
2 | full run |
phase1/roboarc/fiber_bundling_10/ |
2 | full run |
phase1/roboarc/hedgehog_in_cage_05/ |
2 | full run |
phase1/roboarc/hedgehog_in_cage_06/ |
2 | full run |
phase1/roboarc/hedgehog_in_cage_07/ |
2 | full run |
phase1/roboarc/horseshoe_ring_02/ |
2 | full run |
phase1/roboarc/horseshoe_ring_03/ |
2 | full run |
phase1/roboarc/interlocked_claws_05/ |
2 | full run |
phase1/roboarc/interlocked_claws_09/ |
2 | full run |
phase1/roboarc/interlocked_claws_11/ |
2 | full run |
phase1/roboarc/interlocked_claws_12/ |
2 | full run |
phase1/roboarc/knot_untying_06/ |
2 | full run |
phase1/roboarc/knot_untying_09/ |
2 | full run |
phase1/roboarc/lock_picking_04/ |
2 | full run |
phase1/roboarc/lock_picking_05/ |
2 | full run |
phase1/roboarc/rope_threading_03/ |
2 | full run |
phase1/roboarc/rope_threading_04/ |
2 | full run |
phase1/roboarc/rope_threading_05/ |
2 | full run |
phase1/roboarc/rope_threading_06/ |
2 | full run |
phase1/roboarc/rope_threading_07/ |
2 | full run |
phase1/roboarc/rope_threading_08/ |
2 | full run |
phase1/roboarc/slotted_post_ring_04/ |
2 | full run |
phase1/roboarc/slotted_post_ring_05/ |
2 | full run |
phase1/robocasa/turn_off_sink_faucet/ |
2 | full run |
phase1/robocasa365/assemble_cooking_array/ |
2 | full run |
phase1/robocasa365/cereal_and_bowl/ |
2 | full run |
phase1/robocasa365/microwave_defrost_meat/ |
2 | full run |
phase1/robocasa365/searing_meat/ |
2 | full run |
phase1/robodojo/fill_egg_holder/ |
2 | full run |
phase1/robodojo/hang_mugs/ |
2 | full run |
phase1/robodojo/insert_key/ |
2 | full run |
phase1/robodojo/make_kong/ |
2 | full run |
phase1/robodojo/make_toast/ |
2 | full run |
phase1/robodojo/pack_objects_into_box/ |
2 | full run |
phase1/robodojo/store_laptop_and_headphones/ |
2 | full run |
phase1/robodojo/store_laptop_and_headphones_random/ |
2 | full run |
phase1/robodojo/store_tools_in_toolbox/ |
2 | full run |
phase1/robodojo/swap_blocks/ |
2 | full run |
phase1/robodojo/sweep_blocks_random/ |
2 | full run |
phase1/robolab/bbq_sauce_in_bin/ |
2 | full run |
phase1/robolab/food_packing3_cans/ |
2 | full run |
phase1/robolab/put_two_mugs_on_shelf/ |
2 | full run |
phase1/robolab/utensils_in_mug/ |
2 | full run |
phase1/robolab/white_mug_in_center_of_table/ |
2 | full run |
phase1/robopaint/acrylic_applesoranges/ |
2 | full run |
phase1/robopaint/acrylic_bedroom/ |
2 | full run |
phase1/robopaint/acrylic_bluehorses/ |
2 | full run |
phase1/robopaint/acrylic_greatwave/ |
2 | full run |
phase1/robopaint/acrylic_plumpark/ |
2 | full run |
phase1/robopaint/acrylic_redfuji/ |
2 | full run |
phase1/robopaint/acrylic_suddenshower/ |
2 | full run |
phase1/robopaint/acrylic_sunflowers/ |
2 | full run |
phase1/robopaint/acrylic_vision/ |
2 | full run |
phase1/robopaint/caoshu_beiquelindanshui/ |
2 | full run |
phase1/robopaint/caoshu_siwushaonian/ |
2 | full run |
phase1/robopaint/caoshu_zhen/ |
2 | full run |
phase1/robopaint/oil_monalisa_i01/ |
2 | full run |
phase1/robopaint/oil_starrynight_i01/ |
2 | full run |
phase1/robopaint/xingshu_feng/ |
2 | full run |
phase1/robopaint/xingshu_he/ |
2 | full run |
phase1/robopaint/xingshu_huifenghechang/ |
2 | full run |
phase1/robopaint/xingshu_maolinxiuzhu/ |
2 | full run |
phase1/robopaint/xingshu_xian/ |
2 | full run |
phase1/robopaint/xingshu_yong/ |
2 | full run |
phase1/robotwin-2/hanging_mug/ |
2 | full run |
phase1/robotwin-2/place_bread_skillet/ |
2 | full run |
phase1/robowits/cover_with_lid/ |
2 | full run |
phase1/robowits/stack_bowls/ |
2 | full run |
phase1/robowits/stand_bulb/ |
2 | full run |
phase1/robowits/stand_pages/ |
2 | full run |
phase1/vlabench/texas_holdem/ |
2 | full run |
phase1/yamlab/put_pot_on_cooktop/ |
2 | full run |
Phase 2
- Prompts: v3.2 (the frozen labelling prompts with the fixes from a blind recheck of 30 phase-1 tasks; taxonomy v2;
schemas
efe-task-dag/1andefe-trajectory-labels/1). - Writer: Codex
gpt-6.1-sol(effort high for task DAGs and trajectories). Reviewer: Claudeclaude-sonnet-5-5; up to two fix rounds plus a confirm round; files still disputed after that carryreview.status: "disputed". - Task repository: embodied-first-exam commit
80cac7a5a227db930c659c3307a1e79c2a84bc53. - Scope: every benchmark task except the roboengineer suite (434 tasks), two runs per task (one success when there is one, plus a failure; runs with a video first; seeded pick), 868 runs. Every task gets a newly written DAG, and all its phase-2 runs (including runs already labelled in phase 1) are labelled against that DAG.
- Remaining runs (added from 2026-10-29): every other run of each in-scope task (6,648 runs in all) is labelled
against the task's phase-2 DAG as published here (the DAG is not rewritten), with the same prompts, models and
efforts, and added to the same
trajectories/folder; the run that served as a task's reference success (oracle) when its DAG was written stays that task's oracle. Runs without a graded video are labelled from the log only. Status: 5,780 of the 5,780 remaining runs labelled and uploaded (complete). - Run catalog: built from the run index of the dataset
MLL-Lab/embodied-exam-runs(the run viewer'srun.jsonfiles); videos as in phase 1 fromMLL-Lab/embodied-exam-data,runs/<runId>/replay.mp4. Some picked runs and tasks are newer than the catalog shipped with the labelling interface, so importing them needs a catalog refreshed from that run index. - The importer prefers the highest phase: when a task is under
phase1/andphase2/, onlyphase2/<suite>/<task>/(its DAG and its own trajectories) is used;phase1/stays as a record. - Layout as in phase 1:
phase2/<suite>/<task>/task_dag.jsonandphase2/<suite>/<task>/trajectories/<runId>.json, provenance work-directory paths relative,dag_sha256= sha256 oftask_dag.jsonas published here. A task is uploaded whole (its DAG together with all its phase-2 trajectories); remaining-run trajectories are added batch by batch once the batch passes validation. - Phase 2 complete: every task of the run passed validation and is listed below. Status: 434 tasks, 868 trajectories. With the remaining runs: 6,648 trajectories in all.
Phase 2 tasks
| folder | trajectories | also in phase 1 |
|---|---|---|
phase2/ambench/cabinet_pick_place/ |
15 | yes |
phase2/ambench/peg_in_hole/ |
15 | no |
phase2/behavior-1k/assembling_gift_baskets/ |
16 | yes |
phase2/behavior-1k/boxing_books_up_for_storage/ |
16 | yes |
phase2/behavior-1k/boxing_food_after_dinner/ |
15 | yes |
phase2/behavior-1k/bringing_in_wood/ |
16 | yes |
phase2/behavior-1k/bringing_paper_to_recycling/ |
16 | no |
phase2/behavior-1k/bringing_water/ |
16 | yes |
phase2/behavior-1k/can_meat/ |
15 | yes |
phase2/behavior-1k/canning_food/ |
15 | yes |
phase2/behavior-1k/carrying_in_groceries/ |
16 | yes |
phase2/behavior-1k/carrying_out_garden_furniture/ |
16 | no |
phase2/behavior-1k/chopping_wood/ |
16 | yes |
phase2/behavior-1k/clean_a_patio/ |
16 | yes |
phase2/behavior-1k/clean_a_trumpet/ |
15 | yes |
phase2/behavior-1k/clean_boxing_gloves/ |
16 | no |
phase2/behavior-1k/clean_up_your_desk/ |
16 | yes |
phase2/behavior-1k/clean_your_rusty_garden_tools/ |
16 | yes |
phase2/behavior-1k/cleaning_up_branches_and_twigs/ |
16 | yes |
phase2/behavior-1k/cleaning_up_plates_and_food/ |
14 | yes |
phase2/behavior-1k/clearing_food_from_table_into_fridge/ |
15 | yes |
phase2/behavior-1k/collecting_aluminum_cans/ |
16 | yes |
phase2/behavior-1k/collecting_childrens_toys/ |
16 | yes |
phase2/behavior-1k/cook_a_brisket/ |
16 | yes |
phase2/behavior-1k/cook_a_frozen_pie/ |
15 | yes |
phase2/behavior-1k/cook_bacon/ |
16 | yes |
phase2/behavior-1k/cook_broccolini/ |
16 | yes |
phase2/behavior-1k/cook_brussels_sprouts/ |
15 | yes |
phase2/behavior-1k/cook_cabbage/ |
16 | yes |
phase2/behavior-1k/cook_hot_dogs/ |
14 | yes |
phase2/behavior-1k/dispose_of_batteries/ |
16 | yes |
phase2/behavior-1k/freeze_fruit/ |
14 | yes |
phase2/behavior-1k/freeze_pies/ |
16 | yes |
phase2/behavior-1k/halve_an_egg/ |
15 | no |
phase2/behavior-1k/installing_a_fax_machine/ |
15 | no |
phase2/behavior-1k/laying_tile_floors/ |
15 | yes |
phase2/behavior-1k/loading_the_car/ |
14 | yes |
phase2/behavior-1k/make_gift_bags_for_baby_showers/ |
16 | yes |
phase2/behavior-1k/make_pizza/ |
15 | yes |
phase2/behavior-1k/make_rose_centerpieces/ |
14 | yes |
phase2/behavior-1k/organizing_art_supplies/ |
13 | yes |
phase2/behavior-1k/organizing_school_stuff/ |
16 | yes |
phase2/behavior-1k/outfit_a_basic_toolbox/ |
14 | yes |
phase2/behavior-1k/packing_meal_for_delivery/ |
14 | yes |
phase2/behavior-1k/polishing_shoes/ |
13 | no |
phase2/behavior-1k/preparing_lunch_box/ |
13 | yes |
phase2/behavior-1k/put_together_a_basic_pruning_kit/ |
16 | yes |
phase2/behavior-1k/putting_away_toys/ |
15 | yes |
phase2/behavior-1k/putting_dirty_dishes_in_sink/ |
14 | yes |
phase2/behavior-1k/putting_dishes_away_after_cleaning/ |
16 | yes |
phase2/behavior-1k/putting_up_Christmas_decorations_inside/ |
16 | yes |
phase2/behavior-1k/re_shelving_library_books/ |
16 | yes |
phase2/behavior-1k/rearrange_your_room/ |
15 | yes |
phase2/behavior-1k/rearranging_kitchen_furniture/ |
12 | yes |
phase2/behavior-1k/setting_the_fire/ |
16 | yes |
phase2/behavior-1k/setting_the_table/ |
15 | yes |
phase2/behavior-1k/setup_a_bar_for_a_cocktail_party/ |
16 | yes |
phase2/behavior-1k/slicing_vegetables/ |
15 | yes |
phase2/behavior-1k/sorting_books_on_shelf/ |
14 | yes |
phase2/behavior-1k/sorting_bottles_cans_and_paper/ |
14 | yes |
phase2/behavior-1k/sorting_household_items/ |
15 | yes |
phase2/behavior-1k/sorting_vegetables/ |
15 | yes |
phase2/behavior-1k/store_batteries/ |
14 | no |
phase2/behavior-1k/store_produce/ |
15 | no |
phase2/behavior-1k/storing_food/ |
15 | yes |
phase2/behavior-1k/sweeping_garage/ |
14 | yes |
phase2/behavior-1k/thawing_frozen_food/ |
15 | no |
phase2/behavior-1k/tidying_living_room/ |
14 | yes |
phase2/behavior-1k/turning_out_all_lights_before_sleep/ |
15 | no |
phase2/behavior-1k/unloading_the_car/ |
15 | yes |
phase2/behavior-1k/vacuuming_floors/ |
15 | yes |
phase2/behavior-1k/wash_a_baseball_cap/ |
14 | yes |
phase2/behavior-1k/wash_dog_toys/ |
14 | yes |
phase2/bigym2/dishwasher_close/ |
22 | yes |
phase2/bigym2/dishwasher_load_plates/ |
24 | yes |
phase2/bigym2/flip_cutlery/ |
24 | yes |
phase2/bigym2/move_two_plates/ |
24 | yes |
phase2/bigym2/pick_box/ |
24 | yes |
phase2/bigym2/put_cups/ |
24 | yes |
phase2/bigym2/sandwich_remove/ |
24 | yes |
phase2/bigym2/saucepan_to_hob/ |
24 | yes |
phase2/bigym2/stack_blocks/ |
24 | no |
phase2/bigym2/wall_cupboard_open/ |
24 | yes |
phase2/dexhand/assemble_hamburger/ |
16 | yes |
phase2/dexhand/batter_whisk/ |
16 | yes |
phase2/dexhand/bicycle_pump_stroke/ |
16 | yes |
phase2/dexhand/binder_page_load/ |
16 | yes |
phase2/dexhand/book_page_turn/ |
16 | no |
phase2/dexhand/boot_sole_brush/ |
16 | yes |
phase2/dexhand/drying_rack_unfold/ |
16 | yes |
phase2/dexhand/envelope_letter_insert/ |
16 | yes |
phase2/dexhand/essential_cube_reorient/ |
16 | no |
phase2/dexhand/essential_pen_regrasp/ |
16 | yes |
phase2/dexhand/essential_rotary_gait/ |
10 | yes |
phase2/dexhand/essential_two_object/ |
16 | no |
phase2/dexhand/fan_guard_install/ |
16 | yes |
phase2/dexhand/grinder_crank/ |
16 | no |
phase2/dexhand/jar_lid_open/ |
15 | no |
phase2/dexhand/pancake_serve/ |
16 | no |
phase2/dexhand/picture_hang/ |
16 | yes |
phase2/dexhand/ram_module_install/ |
16 | no |
phase2/dexhand/remote_battery_swap/ |
16 | no |
phase2/dexhand/stool_brace_insert/ |
16 | yes |
phase2/dexhand/sweep_dustpan/ |
16 | yes |
phase2/dexhand/usb_plug_insert/ |
16 | yes |
phase2/dexhand/vacuum_bin_reseat/ |
16 | yes |
phase2/dexjoco/bimanual_photograph/ |
16 | yes |
phase2/dexjoco/fold_glasses/ |
16 | no |
phase2/dextoolbench/brush_blue_brush_sweep_forward/ |
16 | yes |
phase2/dextoolbench/brush_red_brush_sweep_forward/ |
16 | yes |
phase2/dextoolbench/brush_red_brush_sweep_right/ |
16 | yes |
phase2/dextoolbench/eraser_handle_eraser_wipe_c/ |
15 | yes |
phase2/dextoolbench/hammer_claw_hammer_swing_down/ |
16 | yes |
phase2/dextoolbench/hammer_claw_hammer_swing_side/ |
16 | yes |
phase2/dextoolbench/hammer_mallet_hammer_swing_down/ |
16 | no |
phase2/dextoolbench/hammer_mallet_hammer_swing_side/ |
15 | yes |
phase2/dextoolbench/marker_sharpie_marker_draw_smile/ |
16 | yes |
phase2/dextoolbench/marker_sharpie_marker_write_c/ |
16 | yes |
phase2/dextoolbench/marker_staples_marker_draw_smile/ |
16 | yes |
phase2/dextoolbench/marker_staples_marker_write_c/ |
16 | yes |
phase2/dextoolbench/screwdriver_long_screwdriver_spin_horizontal/ |
16 | yes |
phase2/dextoolbench/screwdriver_long_screwdriver_spin_vertical/ |
16 | yes |
phase2/dextoolbench/screwdriver_short_screwdriver_spin_horizontal/ |
16 | no |
phase2/dextoolbench/screwdriver_short_screwdriver_spin_vertical/ |
16 | yes |
phase2/dextoolbench/spatula_flat_spatula_flip_over/ |
16 | yes |
phase2/dextoolbench/spatula_flat_spatula_serve_plate/ |
16 | yes |
phase2/dextoolbench/spatula_spoon_spatula_flip_over/ |
16 | yes |
phase2/dextoolbench/spatula_spoon_spatula_serve_plate/ |
16 | yes |
phase2/embodied-olympics-fun/alpine_slalom/ |
16 | no |
phase2/embodied-olympics-fun/backflip/ |
16 | no |
phase2/embodied-olympics-fun/badminton_serve/ |
16 | yes |
phase2/embodied-olympics-fun/balance_beam/ |
15 | no |
phase2/embodied-olympics-fun/balance_board/ |
16 | yes |
phase2/embodied-olympics-fun/basketball_dribble/ |
16 | no |
phase2/embodied-olympics-fun/bowling/ |
16 | no |
phase2/embodied-olympics-fun/box_jump/ |
15 | yes |
phase2/embodied-olympics-fun/boxing/ |
16 | no |
phase2/embodied-olympics-fun/catch/ |
24 | no |
phase2/embodied-olympics-fun/catch_and_shoot/ |
24 | no |
phase2/embodied-olympics-fun/clean_and_jerk/ |
16 | yes |
phase2/embodied-olympics-fun/discus/ |
16 | no |
phase2/embodied-olympics-fun/dodgeball/ |
24 | no |
phase2/embodied-olympics-fun/dribble_slalom/ |
16 | no |
phase2/embodied-olympics-fun/dunk/ |
16 | no |
phase2/embodied-olympics-fun/dunk_between_legs/ |
16 | yes |
phase2/embodied-olympics-fun/free_kick_wall/ |
15 | no |
phase2/embodied-olympics-fun/free_throw/ |
16 | no |
phase2/embodied-olympics-fun/freeski_360/ |
16 | no |
phase2/embodied-olympics-fun/goalkeeper_save/ |
24 | no |
phase2/embodied-olympics-fun/golf_putt/ |
16 | no |
phase2/embodied-olympics-fun/high_bar_swing/ |
16 | no |
phase2/embodied-olympics-fun/high_jump/ |
16 | yes |
phase2/embodied-olympics-fun/hurdles_freestanding/ |
16 | yes |
phase2/embodied-olympics-fun/javelin/ |
16 | no |
phase2/embodied-olympics-fun/juggle_three/ |
24 | no |
phase2/embodied-olympics-fun/juggling/ |
16 | no |
phase2/embodied-olympics-fun/juggling_feet/ |
16 | no |
phase2/embodied-olympics-fun/juggling_head/ |
24 | yes |
phase2/embodied-olympics-fun/jump_rope/ |
16 | no |
phase2/embodied-olympics-fun/layup/ |
16 | no |
phase2/embodied-olympics-fun/long_jump/ |
16 | no |
phase2/embodied-olympics-fun/long_rope/ |
23 | yes |
phase2/embodied-olympics-fun/paddle_bounce/ |
24 | no |
phase2/embodied-olympics-fun/pitch_target/ |
16 | no |
phase2/embodied-olympics-fun/platform_10m/ |
14 | no |
phase2/embodied-olympics-fun/pole_vault/ |
16 | yes |
phase2/embodied-olympics-fun/rugby_conversion/ |
16 | no |
phase2/embodied-olympics-fun/shot_put/ |
15 | no |
phase2/embodied-olympics-fun/slalom/ |
16 | no |
phase2/embodied-olympics-fun/springboard_3m/ |
16 | no |
phase2/embodied-olympics-fun/sprint/ |
16 | no |
phase2/embodied-olympics-fun/standing_long_jump/ |
15 | yes |
phase2/embodied-olympics-fun/steeplechase/ |
16 | yes |
phase2/embodied-olympics-fun/table_tennis_return/ |
23 | no |
phase2/embodied-olympics-fun/taekwondo_kick/ |
15 | no |
phase2/embodied-olympics-fun/tennis_return/ |
23 | no |
phase2/embodied-olympics-fun/tennis_serve/ |
16 | no |
phase2/embodied-olympics-fun/toes_to_bar/ |
16 | no |
phase2/embodied-olympics-fun/trampoline/ |
16 | no |
phase2/embodied-olympics-fun/triple_jump/ |
16 | yes |
phase2/embodied-olympics-fun/vault/ |
16 | yes |
phase2/embodied-olympics-fun/volleyball_serve/ |
16 | no |
phase2/embodied-olympics-fun/volleyball_spike/ |
24 | yes |
phase2/embodied-olympics-fun/wall_ball/ |
16 | no |
phase2/embodied-olympics-match/badminton_duelrl_i01/ |
8 | yes |
phase2/embodied-olympics-match/badminton_match_i01/ |
16 | yes |
phase2/embodied-olympics-match/badminton_match_i02/ |
16 | yes |
phase2/embodied-olympics-match/badminton_match_i04/ |
8 | yes |
phase2/embodied-olympics-match/badminton_matchlow_i01/ |
7 | yes |
phase2/embodied-olympics-match/fencing_bout_i01/ |
7 | yes |
phase2/embodied-olympics-match/foosball_match_i01/ |
8 | yes |
phase2/embodied-olympics-match/soccer_jointcpu_i01/ |
8 | no |
phase2/embodied-olympics-match/soccer_jointgpu_i01/ |
7 | yes |
phase2/embodied-olympics-match/soccer_match/ |
16 | yes |
phase2/embodied-olympics-match/soccerssl_match/ |
15 | yes |
phase2/embodied-olympics-match/soccerssl_match_i01/ |
8 | yes |
phase2/embodied-olympics-match/tabletennis_duelrl_i01/ |
8 | yes |
phase2/embodied-olympics-match/tabletennis_match_i01/ |
16 | yes |
phase2/embodied-olympics-match/tabletennis_match_i02/ |
16 | yes |
phase2/embodied-olympics-match/tabletennis_match_i03/ |
8 | yes |
phase2/embodied-olympics-match/tabletennis_match_i04/ |
8 | no |
phase2/embodied-olympics-match/tabletennis_matchlow_i01/ |
7 | yes |
phase2/embodied-safety-exam/a01_i01/ |
8 | yes |
phase2/embodied-safety-exam/a01_i02/ |
8 | yes |
phase2/embodied-safety-exam/a01_i04/ |
8 | no |
phase2/embodied-safety-exam/a01_i08/ |
8 | yes |
phase2/embodied-safety-exam/a02_i03/ |
8 | yes |
phase2/embodied-safety-exam/a04/ |
8 | yes |
phase2/embodied-safety-exam/a06_i01/ |
8 | yes |
phase2/embodied-safety-exam/a06_i03/ |
8 | yes |
phase2/embodied-safety-exam/a08_i01/ |
8 | yes |
phase2/embodied-safety-exam/a17/ |
8 | yes |
phase2/embodied-safety-exam/a26_i02/ |
8 | yes |
phase2/embodied-safety-exam/a27_i06/ |
8 | yes |
phase2/embodied-safety-exam/a28/ |
8 | yes |
phase2/embodied-safety-exam/a28_i01/ |
8 | yes |
phase2/embodied-safety-exam/a28_i03/ |
8 | yes |
phase2/embodied-safety-exam/a29/ |
8 | no |
phase2/embodied-safety-exam/a29_i01/ |
8 | yes |
phase2/embodied-safety-exam/a29_i03/ |
8 | yes |
phase2/embodied-safety-exam/a33_i01/ |
8 | yes |
phase2/embodied-safety-exam/a33_i04/ |
8 | yes |
phase2/embodied-safety-exam/a34_i07/ |
8 | yes |
phase2/embodied-safety-exam/a34_i09/ |
8 | yes |
phase2/embodied-safety-exam/a34_i11/ |
8 | yes |
phase2/embodied-safety-exam/b06_i01/ |
8 | yes |
phase2/embodied-safety-exam/b06_i02/ |
8 | yes |
phase2/embodied-safety-exam/b11/ |
8 | yes |
phase2/embodied-safety-exam/b11_i01/ |
7 | yes |
phase2/embodied-safety-exam/b11_i03/ |
8 | yes |
phase2/embodied-safety-exam/b12_i04/ |
8 | yes |
phase2/embodied-safety-exam/b12_i06/ |
8 | yes |
phase2/embodied-safety-exam/b12_i07/ |
8 | yes |
phase2/embodied-safety-exam/c02/ |
16 | yes |
phase2/embodied-safety-exam/c02_i01/ |
16 | no |
phase2/embodied-safety-exam/c04_i01/ |
15 | no |
phase2/embodied-safety-exam/c05_i01/ |
16 | yes |
phase2/embodied-safety-exam/c05_i02/ |
8 | yes |
phase2/embodied-safety-exam/c16/ |
8 | yes |
phase2/embodied-safety-exam/d12_i04/ |
8 | yes |
phase2/embodied-safety-exam/e06/ |
8 | yes |
phase2/embodied-safety-exam/e06_i03/ |
8 | yes |
phase2/embodied-safety-exam/f01/ |
8 | yes |
phase2/embodied-safety-exam/f04_i01/ |
8 | yes |
phase2/embodied-safety-exam/f04_i02/ |
8 | yes |
phase2/embodied-safety-exam/f05/ |
8 | yes |
phase2/embodied-safety-exam/f05_i01/ |
8 | yes |
phase2/embodied-safety-exam/f06/ |
8 | yes |
phase2/embodied-safety-exam/f06_i01/ |
8 | yes |
phase2/embodied-safety-exam/f08/ |
8 | yes |
phase2/embodied-safety-exam/f09/ |
8 | yes |
phase2/embodied-safety-exam/f11/ |
8 | yes |
phase2/humanoidbench/loco_balance_hard/ |
23 | yes |
phase2/humanoidbench/loco_hurdle/ |
24 | yes |
phase2/humanoidbench/manip_bookshelf_simple/ |
24 | yes |
phase2/humanoidbench/manip_cube/ |
24 | no |
phase2/humanoidbench/manip_door/ |
23 | yes |
phase2/humanoidbench/manip_powerlift/ |
24 | yes |
phase2/humanoidbench/manip_room/ |
24 | yes |
phase2/humanoidbench/manip_window/ |
24 | yes |
phase2/ispatialgym/alignment_axial/ |
16 | yes |
phase2/ispatialgym/camera_relocalization_obstructed/ |
16 | yes |
phase2/ispatialgym/configuration_matching_articulation_open_small_scale/ |
16 | yes |
phase2/ispatialgym/configuration_matching_block_design/ |
15 | yes |
phase2/ispatialgym/inter_object_relation_satisfaction_nextto_small_scale/ |
15 | yes |
phase2/ispatialgym/inter_object_relation_satisfaction_ontop_small_scale/ |
16 | yes |
phase2/ispatialgym/inter_object_relation_satisfaction_touching_small_scale/ |
16 | yes |
phase2/ispatialgym/mental_rotation/ |
15 | no |
phase2/kinder/constrained_cupboard/ |
16 | no |
phase2/kinder/scoop_pour_i01/ |
16 | yes |
phase2/kinder/sweep_into_drawer/ |
16 | yes |
phase2/kinder/sweep_simple_i01/ |
16 | yes |
phase2/libero-max/goal_akita_black_bowl_in_wooden_cabinet_top_target_relocation/ |
16 | no |
phase2/libero-max/goal_wine_bottle_on_wine_rack_top_visual_theme_switch/ |
16 | yes |
phase2/maniguard/cabinet_pickup_i22/ |
16 | yes |
phase2/maniguard/cabinet_pickup_i26/ |
16 | yes |
phase2/maniguard/dusty_transfer_i03/ |
16 | yes |
phase2/maniguard/dusty_transfer_i16/ |
16 | yes |
phase2/maniguard/jar_transport_i13/ |
16 | yes |
phase2/maniguard/jar_transport_i25/ |
16 | no |
phase2/maniguard/lid_transport_i12/ |
16 | yes |
phase2/metaworldplus/basketball/ |
16 | no |
phase2/metaworldplus/button_press_topdown/ |
15 | yes |
phase2/molmospaces/navigate_to/ |
16 | no |
phase2/molmospaces/pick/ |
16 | yes |
phase2/mujoco-playground/aloha_single_peg_insertion/ |
16 | yes |
phase2/mujoco-playground/barkour_joystick/ |
24 | yes |
phase2/mujoco-playground/go1_handstand/ |
16 | yes |
phase2/mujoco-playground/go1_joystick_flat_terrain/ |
16 | yes |
phase2/mujoco-playground/go1_joystick_rough_terrain/ |
16 | no |
phase2/mujoco-playground/spot_flat_terrain_joystick/ |
16 | yes |
phase2/mujoco-playground/spot_joystick_gait_tracking/ |
16 | yes |
phase2/myosuite/baoding/ |
16 | yes |
phase2/myosuite/bimanual/ |
16 | yes |
phase2/myosuite/chase_tag/ |
16 | yes |
phase2/myosuite/dex_airplane_fly/ |
16 | yes |
phase2/myosuite/dex_apple_lift/ |
16 | yes |
phase2/myosuite/dex_bunny_inspect/ |
16 | yes |
phase2/myosuite/dex_cup_pour/ |
16 | yes |
phase2/myosuite/dex_cylinder_pass/ |
16 | yes |
phase2/myosuite/dex_hammer_use/ |
16 | no |
phase2/myosuite/die_reorient/ |
16 | yes |
phase2/myosuite/obj_reorient/ |
16 | yes |
phase2/myosuite/osl_run/ |
16 | yes |
phase2/myosuite/soccer/ |
16 | yes |
phase2/pandadyn/d01_reciprocal_cup_relay/ |
24 | yes |
phase2/pandadyn/d03_cooperative_puck_course/ |
23 | yes |
phase2/pandadyn/d05_moving_ramp_relay/ |
16 | no |
phase2/pandadyn/d07_spring_gate_switchyard/ |
24 | yes |
phase2/pandadyn/d09_lever_launch_catch_reload/ |
24 | yes |
phase2/pandadyn/s01_cup_route_delivery/ |
24 | yes |
phase2/pandadyn/s06_cube_flip_route_recover/ |
16 | yes |
phase2/pandadyn/s09_toss_recatch_delivery/ |
23 | yes |
phase2/pandadyn/s11_cylinder_rescue_route_deliver/ |
24 | yes |
phase2/pandadyn/s12_moving_bridge_puck_relay/ |
24 | yes |
phase2/roboarc/blanket_flatten_02/ |
14 | yes |
phase2/roboarc/blanket_flatten_03/ |
15 | yes |
phase2/roboarc/blanket_flatten_04/ |
15 | no |
phase2/roboarc/blanket_flatten_07/ |
15 | yes |
phase2/roboarc/blanket_flatten_08/ |
15 | yes |
phase2/roboarc/blanket_fold_01/ |
15 | yes |
phase2/roboarc/blanket_fold_03/ |
15 | yes |
phase2/roboarc/blanket_fold_04/ |
15 | yes |
phase2/roboarc/blanket_fold_05/ |
16 | yes |
phase2/roboarc/buzz_wire_3d_01/ |
16 | yes |
phase2/roboarc/buzz_wire_3d_02/ |
16 | yes |
phase2/roboarc/buzz_wire_3d_04/ |
16 | yes |
phase2/roboarc/buzz_wire_3d_06/ |
16 | yes |
phase2/roboarc/buzz_wire_3d_07/ |
16 | yes |
phase2/roboarc/buzz_wire_3d_09/ |
16 | yes |
phase2/roboarc/buzz_wire_3d_10/ |
16 | yes |
phase2/roboarc/cast_coil_04/ |
15 | yes |
phase2/roboarc/cast_coil_06/ |
16 | yes |
phase2/roboarc/cast_coil_07/ |
16 | no |
phase2/roboarc/cast_coil_08/ |
16 | yes |
phase2/roboarc/cast_keyhole_03/ |
16 | yes |
phase2/roboarc/cast_keyhole_04/ |
16 | yes |
phase2/roboarc/cast_keyhole_05/ |
16 | yes |
phase2/roboarc/fiber_bundling_04/ |
16 | yes |
phase2/roboarc/fiber_bundling_06/ |
16 | yes |
phase2/roboarc/fiber_bundling_07/ |
16 | yes |
phase2/roboarc/fiber_bundling_09/ |
16 | yes |
phase2/roboarc/fiber_bundling_10/ |
16 | yes |
phase2/roboarc/hedgehog_in_cage_05/ |
16 | yes |
phase2/roboarc/hedgehog_in_cage_06/ |
16 | yes |
phase2/roboarc/hedgehog_in_cage_07/ |
16 | yes |
phase2/roboarc/horseshoe_ring_02/ |
16 | yes |
phase2/roboarc/horseshoe_ring_03/ |
16 | yes |
phase2/roboarc/ikea_eket/ |
16 | no |
phase2/roboarc/ikea_gladom/ |
16 | no |
phase2/roboarc/ikea_ikea365/ |
16 | no |
phase2/roboarc/ikea_lack/ |
16 | no |
phase2/roboarc/ikea_stegesund/ |
16 | no |
phase2/roboarc/ikea_svallet/ |
16 | no |
phase2/roboarc/ikea_variera/ |
16 | no |
phase2/roboarc/ikea_vesken/ |
16 | no |
phase2/roboarc/interlocked_claws_05/ |
16 | yes |
phase2/roboarc/interlocked_claws_09/ |
15 | yes |
phase2/roboarc/interlocked_claws_11/ |
16 | yes |
phase2/roboarc/interlocked_claws_12/ |
16 | yes |
phase2/roboarc/knot_untying_06/ |
16 | yes |
phase2/roboarc/knot_untying_09/ |
15 | yes |
phase2/roboarc/lock_picking_04/ |
16 | yes |
phase2/roboarc/lock_picking_05/ |
16 | yes |
phase2/roboarc/rope_threading_03/ |
15 | yes |
phase2/roboarc/rope_threading_04/ |
16 | yes |
phase2/roboarc/rope_threading_05/ |
15 | yes |
phase2/roboarc/rope_threading_06/ |
15 | yes |
phase2/roboarc/rope_threading_07/ |
16 | yes |
phase2/roboarc/rope_threading_08/ |
15 | yes |
phase2/roboarc/slotted_post_ring_03/ |
15 | no |
phase2/roboarc/slotted_post_ring_04/ |
16 | yes |
phase2/roboarc/slotted_post_ring_05/ |
16 | yes |
phase2/robocasa-gr1/potato_to_microwave_close/ |
16 | no |
phase2/robocasa/coffee_setup_mug/ |
16 | no |
phase2/robocasa/turn_off_sink_faucet/ |
16 | yes |
phase2/robocasa365/assemble_cooking_array/ |
16 | yes |
phase2/robocasa365/cereal_and_bowl/ |
16 | yes |
phase2/robocasa365/display_meat_variety/ |
16 | no |
phase2/robocasa365/microwave_defrost_meat/ |
16 | yes |
phase2/robocasa365/searing_meat/ |
16 | yes |
phase2/robodojo/fill_egg_holder/ |
16 | yes |
phase2/robodojo/hang_mugs/ |
16 | yes |
phase2/robodojo/hang_mugs_random/ |
15 | no |
phase2/robodojo/insert_key/ |
16 | yes |
phase2/robodojo/make_kong/ |
16 | yes |
phase2/robodojo/make_toast/ |
16 | yes |
phase2/robodojo/pack_objects_into_box/ |
16 | yes |
phase2/robodojo/store_laptop_and_headphones/ |
15 | yes |
phase2/robodojo/store_laptop_and_headphones_random/ |
16 | yes |
phase2/robodojo/store_tools_in_toolbox/ |
16 | yes |
phase2/robodojo/swap_blocks/ |
16 | yes |
phase2/robodojo/sweep_blocks_random/ |
16 | yes |
phase2/robolab/bbq_sauce_in_bin/ |
14 | yes |
phase2/robolab/food_packing3_cans/ |
14 | yes |
phase2/robolab/put_two_mugs_on_shelf/ |
15 | yes |
phase2/robolab/red_dishes_in_bin/ |
13 | no |
phase2/robolab/utensils_in_mug/ |
15 | yes |
phase2/robolab/white_mug_in_center_of_table/ |
16 | yes |
phase2/robopaint/acrylic_applesoranges/ |
16 | yes |
phase2/robopaint/acrylic_bedroom/ |
16 | yes |
phase2/robopaint/acrylic_bluehorses/ |
16 | yes |
phase2/robopaint/acrylic_greatwave/ |
15 | yes |
phase2/robopaint/acrylic_kajikazawa/ |
16 | no |
phase2/robopaint/acrylic_plumpark/ |
16 | yes |
phase2/robopaint/acrylic_redfuji/ |
16 | yes |
phase2/robopaint/acrylic_suddenshower/ |
16 | yes |
phase2/robopaint/acrylic_sunflowers/ |
15 | yes |
phase2/robopaint/acrylic_vision/ |
16 | yes |
phase2/robopaint/caoshu_beiquelindanshui/ |
16 | yes |
phase2/robopaint/caoshu_siwushaonian/ |
16 | yes |
phase2/robopaint/caoshu_zhen/ |
16 | yes |
phase2/robopaint/kaishu_linfengguilong/ |
16 | no |
phase2/robopaint/oil_monalisa_i01/ |
16 | yes |
phase2/robopaint/oil_starrynight_i01/ |
16 | yes |
phase2/robopaint/xingshu_feng/ |
16 | yes |
phase2/robopaint/xingshu_he/ |
16 | yes |
phase2/robopaint/xingshu_huifenghechang/ |
16 | yes |
phase2/robopaint/xingshu_maolinxiuzhu/ |
16 | yes |
phase2/robopaint/xingshu_xian/ |
16 | yes |
phase2/robopaint/xingshu_yong/ |
16 | yes |
phase2/robotwin-2/hanging_mug/ |
16 | yes |
phase2/robotwin-2/place_bread_skillet/ |
16 | yes |
phase2/robotwin-2/put_bottles_dustbin/ |
16 | no |
phase2/robowits/ball_into_bottle/ |
16 | no |
phase2/robowits/cover_with_lid/ |
16 | yes |
phase2/robowits/retrieve_cube/ |
16 | no |
phase2/robowits/stack_bowls/ |
15 | yes |
phase2/robowits/stand_bulb/ |
16 | yes |
phase2/robowits/stand_pages/ |
16 | yes |
phase2/vlabench/heat_food/ |
16 | no |
phase2/vlabench/texas_holdem/ |
16 | yes |
phase2/yamlab/hang_mug_on_tree_i15/ |
16 | no |
phase2/yamlab/put_pot_on_cooktop/ |
16 | yes |
Phase K: Kimi 3.1 Pro (2026-11-08)
- Runs: the counted attempt of Claude Code with Kimi 3.1 Pro (run
claude_code-kimi_3_1_pro-high) on each core test (privileged, standard and realtime), successes and failures alike, the same rule as for the other seven agents. Tests whose task has no phase-2 DAG are not labelled (no DAG is written here): embodiedswe, roboengineer and two newer Embodied Olympics - Match tasks. - Writer: Claude Sonnet 5.5 (
claude-sonnet-5-5, Claude Code) for this agent only, by the owner's decision of 2026-11-08 (no GPT-6.1 Sol quota). The reviewer is Claude Sonnet 5.5 as before, in a separate session. Everything else is unchanged: prompts v3.2, the task's phase-2 DAG as published underphase2/(not rewritten;dag_sha256is that file), the same task inputs and oracle rule, the same review rounds and validation. Each file'sprovenancesaysengine: claude,model: claude-sonnet-5-5and carries awriter_note. - Videos: the graded
replay.mp4of the run viewer datasetMLL-Lab/embodied-exam-runs; runs without one (privileged runs that submitted nothing) are labelled from the log only. - Layout:
phaseK/<suite>/<task>/trajectories/<runId>.json; the DAG staysphase2/<suite>/<task>/task_dag.json. - Status: 857 of 857 trajectories uploaded (427 tasks) (complete).
Phase R: unlabelled counted attempts of the seven original agents (2026-11-08)
- Runs: the counted attempt of each of the seven original agents on each core test of the eight-agent leaderboard
(456 Standard, 391 Privileged, 54 Realtime) whose task has a phase-2 DAG, when that attempt had no label under
phase2/: reruns after 2026-11-06 that replaced a labelled attempt (among them the RoboDojo reruns), and attempts newer than the phase-2 run catalog. Renamed realtime runs (-standard-realtime->-realtime, same trial) count as labelled and are not repeated. - Batch R10 (10 trajectories): ten core cells (six of Qwen3.8 Max, four of GPT-6 Astra, all Standard) have no counted
attempt in the leaderboard; their earlier attempt on the same test is labelled here and used for the cell in the
stage-level analysis only (the success counts are the leaderboard's). One of them (
robodojo-make-kong-i00-standard, GPT-6 Astra) succeeded and is left out of the failure statistics. - Writer: Claude Sonnet 5.5 (
claude-sonnet-5-5, Claude Code), as for phase K, by the owner's decision of 2026-11-08; the reviewer is Claude Sonnet 5.5 in a separate session. Prompts v3.2, the task's phase-2 DAG as published underphase2/(not rewritten), the same task inputs, oracle rule, review rounds and validation. Each file'sprovenancesaysengine: claude,model: claude-sonnet-5-5and carries awriter_note. - Layout:
phaseR/<suite>/<task>/trajectories/<runId>.json; the DAG staysphase2/<suite>/<task>/task_dag.json. - Status: 630 of 630 trajectories uploaded (269 tasks) (complete).
Phase F: Claude Fable 5.2 (2026-11-12)
- Runs: the counted attempt of Claude Code with Claude Fable 5.2 (run
claude_code-fable_5_2-high, default seed) on each core test (privileged, standard and realtime), successes and failures alike, the same rule as for the other agents. Tests whose task has no phase-2 DAG are not labelled (no DAG is written here): embodiedswe, roboengineer and two newer Embodied Olympics - Match tasks. - Writer: GPT-6.1 Sol (
gpt-6.1-sol, high effort) in Codex; reviewer: Claude Sonnet 5.5 in Claude Code. The same writer and reviewer as phase 2; prompts v3.2, the task's phase-2 DAG as published underphase2/(not rewritten;dag_sha256is that file), the same task inputs and oracle rule, review rounds and validation. Each file'sprovenancecarries awriter_noteandwriter_access(ChatGPT subscription or Azure OpenAI API, same model). - Videos: the graded
replay.mp4of the run viewer datasetMLL-Lab/embodied-exam-runs; runs without one (privileged runs that submitted nothing) are labelled from the log only. - Layout:
phaseF/<suite>/<task>/trajectories/<runId>.json; the DAG staysphase2/<suite>/<task>/task_dag.json. - Status: 855 of 855 trajectories uploaded (427 tasks) (complete).
Phase Sol: GPT-6.1 Sol (2026-11-15)
- Runs: the counted attempt of Codex with GPT-6.1 Sol (run
codex-gpt6_1_sol-high, high reasoning effort) on each core test (456 Standard, 391 Privileged, 54 Realtime), successes and failures alike, the same rule as for the other agents. Tests whose task has no phase-2 DAG are not labelled (no DAG is written here): embodiedswe, roboengineer and two newer Embodied Olympics - Match tasks. Five Realtime tests still queued and one Privileged test whose attempt ended in an infrastructure failure have no attempt to label. - Writer: GPT-6.1 Sol (
gpt-6.1-sol, high effort) in Codex; reviewer: Claude Sonnet 5.5 in Claude Code. The same writer and reviewer as phases 2 and F; prompts v3.2, the task's phase-2 DAG as published underphase2/(not rewritten;dag_sha256is that file), the same task inputs and oracle rule, review rounds and validation. Each file'sprovenancecarries awriter_noteandwriter_access(ChatGPT subscription). Here the writer is the model of the labelled agent; the reviewer is a different model. - Videos: this run is not in the run viewer dataset
MLL-Lab/embodied-exam-runs. Its run records were built with the run viewer's builder (same format) from the trials, and the video is the trial's gradedreplay.mp4, the same file the run viewer uses. Runs without one are labelled from the log only (6 runs: 4 RoboWits and 1 DexHand Privileged runs whose replay wrote no video, 1 RoboDojo Standard run that hit the agent time limit and has only a live recording). - Layout:
phaseSol/<suite>/<task>/trajectories/<runId>.json; the DAG staysphase2/<suite>/<task>/task_dag.json. - Status: 852 of 852 trajectories uploaded (427 tasks) (complete).
- Downloads last month
- 556