Dataset Viewer
Duplicate
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code:   StreamingRowsError
Exception:    CastError
Message:      Couldn't cast
schema: string
taxonomy: string
runId: string
taskKey: string
dag_sha256: string
outcome: struct<success: int64, reward: double, source: string, raw: struct<success: int64, all_success: int6 (... 902 chars omitted)
  child 0, success: int64
  child 1, reward: double
  child 2, source: string
  child 3, raw: struct<success: int64, all_success: int64, deterministic: int64, state_identical: int64, determinism (... 841 chars omitted)
      child 0, success: int64
      child 1, all_success: int64
      child 2, deterministic: int64
      child 3, state_identical: int64
      child 4, determinism_checked: int64
      child 5, instance_ok: int64
      child 6, n_actions: int64
      child 7, replays: int64
      child 8, sim_steps: int64
      child 9, sim_time_s: double
      child 10, timed_out: int64
      child 11, replay_wall_s: double
      child 12, replay_timeout_s: double
      child 13, grader_error: int64
      child 14, infra_error: int64
      child 15, replay_retries: int64
      child 16, replay_gpu_fault: int64
      child 17, agent_sessions: int64
      child 18, agent_resets: int64
      child 19, agent_sim_steps: int64
      child 20, agent_records: int64
      child 21, agent_replays: int64
      child 22, agent_cost_self_reported: int64
      child 23, video_rendered: int64
      child 24, video_episode_success: int64
      child 25, resource_enforced: int64
      child 26, limits_violation_observed: int64
      child 27, resource_monitor_missing: int64
      c
...
hild 9, step: struct<cat: string, item: string>
              child 0, cat: string
              child 1, item: string
          child 10, contact: list<item: struct<who: string, how: string, note_zh: string, plan: int64>>
              child 0, item: struct<who: string, how: string, note_zh: string, plan: int64>
                  child 0, who: string
                  child 1, how: string
                  child 2, note_zh: string
                  child 3, plan: int64
          child 11, optional: bool
          child 12, verify_zh: null
          child 13, sources: list<item: string>
              child 0, item: string
          child 14, conf: int64
          child 15, note_zh: string
  child 6, constraints: list<item: null>
      child 0, item: null
  child 7, success: null
  child 8, atoms: list<item: struct<atom: string, level: string, source: string>>
      child 0, item: struct<atom: string, level: string, source: string>
          child 0, atom: string
          child 1, level: string
          child 2, source: string
task_class: string
suite: string
evidence_check: struct<n_state_runs: int64, n_faithful: int64, success_runs_reaching_all: string, spec_success_eq_gr (... 76 chars omitted)
  child 0, n_state_runs: int64
  child 1, n_faithful: int64
  child 2, success_runs_reaching_all: string
  child 3, spec_success_eq_grader: string
  child 4, diverged_runs: list<item: null>
      child 0, item: null
  child 5, notes_zh: list<item: string>
      child 0, item: string
to
{'schema': Value('string'), 'taxonomy': Value('string'), 'taskKey': Value('string'), 'suite': Value('string'), 'base': Value('string'), 'title_zh': Value('string'), 'summary_zh': Value('string'), 'embodiment_zh': Value('string'), 'task_class': Value('string'), 'analysis_level': Value('string'), 'spec': {'version': Value('string'), 'task': Value('string'), 'bench': Value('string'), 'progress': Value('string'), 'entities': {}, 'milestones': List({'id': Value('string'), 'zh': Value('string'), 'mode': Value('string'), 'when': Value('null'), 'after': List(Value('string')), 'hold_s': Value('int64'), 'freeze_held': Value('bool'), 'goal': Value('bool'), 'criterion_zh': Value('string'), 'step': {'cat': Value('string'), 'item': Value('string')}, 'contact': List({'who': Value('string'), 'how': Value('string'), 'note_zh': Value('string'), 'plan': Value('int64')}), 'optional': Value('bool'), 'verify_zh': Value('null'), 'sources': List(Value('string')), 'conf': Value('int64'), 'note_zh': Value('string')}), 'constraints': List(Value('null')), 'success': Value('null'), 'atoms': List({'atom': Value('string'), 'level': Value('string'), 'source': Value('string')})}, 'native_metrics': List({'name': Value('string'), 'unit': Value('string'), 'threshold': Value('float64'), 'direction': Value('string'), 'note_zh': Value('string')}), 'open_questions_zh': List(Value('string')), 'conf': Value('int64'), 'conf_reason_zh': Value('string'), 'review': {'status': Value('string'), 'rounds': Value('int64'), 'reviewers': List({'engine': Value('string'), 'model': Value('string'), 'round': Value('int64'), 'verdict': Value('string'), 'conf': Value('int64'), 'summary_zh': Value('string'), 'n_issues': Value('int64'), 'n_major': Value('int64')}), 'open_zh': List(Value('null'))}, 'evidence_check': {'n_state_runs': Value('int64'), 'n_faithful': Value('int64'), 'success_runs_reaching_all': Value('string'), 'spec_success_eq_grader': Value('string'), 'diverged_runs': List(Value('null')), 'notes_zh': List(Value('string'))}, 'provenance': {'prompt': Value('string'), 'prompt_sha256': Value('string'), 'engine': Value('string'), 'model': Value('string'), 'effort': Value('string'), 'created': Value('timestamp[s]'), 'review_prompt': Value('string'), 'review_prompt_sha256': Value('string'), 'reconcile_prompt': Value('string'), 'reconcile_prompt_sha256': Value('string'), 'calls': List({'stage': Value('string'), 'round': Value('int64'), 'ok': Value('bool'), 'space': Value('string')}), 'review_engine': Value('string'), 'review_model': Value('string'), 'inputs_manifest_sha256': Value('string'), 'repo_commit': Value('string')}}
because column names don't match
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/utils.py", line 149, in get_rows_or_raise
                  return get_rows(
                      dataset=dataset,
                  ...<4 lines>...
                      column_names=column_names,
                  )
                File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
                  return func(*args, **kwargs)
                File "/src/services/worker/src/worker/utils.py", line 129, in get_rows
                  rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
                File "/src/services/worker/src/worker/utils.py", line 544, in safe_iter
                  yield from ds.decode(False) if ds.features else ds
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 3052, in __iter__
                  for key, example in ex_iterable:
                                      ^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2559, in __iter__
                  for key, pa_table in self._iter_arrow():
                                       ~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2584, in _iter_arrow
                  for key, pa_table in self.ex_iterable._iter_arrow():
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 558, in _iter_arrow
                  for key, pa_table in iterator:
                                       ^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 440, in _iter_arrow
                  for key, pa_table in self.generate_tables_fn(**gen_kwags):
                                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 361, in _generate_tables
                  self._cast_table(pa_table, json_field_paths=json_field_paths),
                  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
                  pa_table = table_cast(pa_table, self.info.features.arrow_schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2381, in table_cast
                  return cast_table_to_schema(table, schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2309, in cast_table_to_schema
                  raise CastError(
                  ...<3 lines>...
                  )
              datasets.table.CastError: Couldn't cast
              schema: string
              taxonomy: string
              runId: string
              taskKey: string
              dag_sha256: string
              outcome: struct<success: int64, reward: double, source: string, raw: struct<success: int64, all_success: int6 (... 902 chars omitted)
                child 0, success: int64
                child 1, reward: double
                child 2, source: string
                child 3, raw: struct<success: int64, all_success: int64, deterministic: int64, state_identical: int64, determinism (... 841 chars omitted)
                    child 0, success: int64
                    child 1, all_success: int64
                    child 2, deterministic: int64
                    child 3, state_identical: int64
                    child 4, determinism_checked: int64
                    child 5, instance_ok: int64
                    child 6, n_actions: int64
                    child 7, replays: int64
                    child 8, sim_steps: int64
                    child 9, sim_time_s: double
                    child 10, timed_out: int64
                    child 11, replay_wall_s: double
                    child 12, replay_timeout_s: double
                    child 13, grader_error: int64
                    child 14, infra_error: int64
                    child 15, replay_retries: int64
                    child 16, replay_gpu_fault: int64
                    child 17, agent_sessions: int64
                    child 18, agent_resets: int64
                    child 19, agent_sim_steps: int64
                    child 20, agent_records: int64
                    child 21, agent_replays: int64
                    child 22, agent_cost_self_reported: int64
                    child 23, video_rendered: int64
                    child 24, video_episode_success: int64
                    child 25, resource_enforced: int64
                    child 26, limits_violation_observed: int64
                    child 27, resource_monitor_missing: int64
                    c
              ...
              hild 9, step: struct<cat: string, item: string>
                            child 0, cat: string
                            child 1, item: string
                        child 10, contact: list<item: struct<who: string, how: string, note_zh: string, plan: int64>>
                            child 0, item: struct<who: string, how: string, note_zh: string, plan: int64>
                                child 0, who: string
                                child 1, how: string
                                child 2, note_zh: string
                                child 3, plan: int64
                        child 11, optional: bool
                        child 12, verify_zh: null
                        child 13, sources: list<item: string>
                            child 0, item: string
                        child 14, conf: int64
                        child 15, note_zh: string
                child 6, constraints: list<item: null>
                    child 0, item: null
                child 7, success: null
                child 8, atoms: list<item: struct<atom: string, level: string, source: string>>
                    child 0, item: struct<atom: string, level: string, source: string>
                        child 0, atom: string
                        child 1, level: string
                        child 2, source: string
              task_class: string
              suite: string
              evidence_check: struct<n_state_runs: int64, n_faithful: int64, success_runs_reaching_all: string, spec_success_eq_gr (... 76 chars omitted)
                child 0, n_state_runs: int64
                child 1, n_faithful: int64
                child 2, success_runs_reaching_all: string
                child 3, spec_success_eq_grader: string
                child 4, diverged_runs: list<item: null>
                    child 0, item: null
                child 5, notes_zh: list<item: string>
                    child 0, item: string
              to
              {'schema': Value('string'), 'taxonomy': Value('string'), 'taskKey': Value('string'), 'suite': Value('string'), 'base': Value('string'), 'title_zh': Value('string'), 'summary_zh': Value('string'), 'embodiment_zh': Value('string'), 'task_class': Value('string'), 'analysis_level': Value('string'), 'spec': {'version': Value('string'), 'task': Value('string'), 'bench': Value('string'), 'progress': Value('string'), 'entities': {}, 'milestones': List({'id': Value('string'), 'zh': Value('string'), 'mode': Value('string'), 'when': Value('null'), 'after': List(Value('string')), 'hold_s': Value('int64'), 'freeze_held': Value('bool'), 'goal': Value('bool'), 'criterion_zh': Value('string'), 'step': {'cat': Value('string'), 'item': Value('string')}, 'contact': List({'who': Value('string'), 'how': Value('string'), 'note_zh': Value('string'), 'plan': Value('int64')}), 'optional': Value('bool'), 'verify_zh': Value('null'), 'sources': List(Value('string')), 'conf': Value('int64'), 'note_zh': Value('string')}), 'constraints': List(Value('null')), 'success': Value('null'), 'atoms': List({'atom': Value('string'), 'level': Value('string'), 'source': Value('string')})}, 'native_metrics': List({'name': Value('string'), 'unit': Value('string'), 'threshold': Value('float64'), 'direction': Value('string'), 'note_zh': Value('string')}), 'open_questions_zh': List(Value('string')), 'conf': Value('int64'), 'conf_reason_zh': Value('string'), 'review': {'status': Value('string'), 'rounds': Value('int64'), 'reviewers': List({'engine': Value('string'), 'model': Value('string'), 'round': Value('int64'), 'verdict': Value('string'), 'conf': Value('int64'), 'summary_zh': Value('string'), 'n_issues': Value('int64'), 'n_major': Value('int64')}), 'open_zh': List(Value('null'))}, 'evidence_check': {'n_state_runs': Value('int64'), 'n_faithful': Value('int64'), 'success_runs_reaching_all': Value('string'), 'spec_success_eq_grader': Value('string'), 'diverged_runs': List(Value('null')), 'notes_zh': List(Value('string'))}, 'provenance': {'prompt': Value('string'), 'prompt_sha256': Value('string'), 'engine': Value('string'), 'model': Value('string'), 'effort': Value('string'), 'created': Value('timestamp[s]'), 'review_prompt': Value('string'), 'review_prompt_sha256': Value('string'), 'reconcile_prompt': Value('string'), 'reconcile_prompt_sha256': Value('string'), 'calls': List({'stage': Value('string'), 'round': Value('int64'), 'ok': Value('bool'), 'space': Value('string')}), 'review_engine': Value('string'), 'review_model': Value('string'), 'inputs_manifest_sha256': Value('string'), 'repo_commit': Value('string')}}
              because column names don't match

Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.

EFE failure analysis labels

Failure-analysis labels for runs of the Embodied First Exam (EFE) benchmark. Per task: a task DAG (semantic steps, step types, contact, prerequisite edges). Per run (trajectory): when each step holds on the run's video (progress map), the contact that actually happened, and for a failed run the breakpoint step, the failure phenomena and the root causes. Label text is in Chinese.

The labels come in two phases: phase 1 (folder phase1/, prompts v3.1, sections below) and phase 2 (folder phase2/, prompts v3.2, section Phase 2 at the end). For a task in both folders, use phase 2: the importer reads only the highest phaseN/ folder of a task (its DAG with its own trajectories).

  • Prompts: v3.1 (frozen labelling prompts, taxonomy v2; schemas efe-task-dag/1 and efe-trajectory-labels/1).
  • Writer: Codex gpt-6.1-sol (effort high for task DAGs and trajectories). Reviewer: Claude claude-sonnet-5-5; up to two fix rounds plus a confirm round; files still disputed after that carry review.status: "disputed".
  • Task repository: embodied-first-exam commit 4c6d01b9990e3b78ddb620bbd237932b3696f60e.
  • Phase 1: two runs per task (one success when there is one, plus a failure of another model; runs with a video first). Run records and videos: dataset MLL-Lab/embodied-exam-data, runs/<runId>.
  • Phase 1 complete: every task of the run passed validation and is listed below. Status: 331 tasks (3 pilot, 328 full run), 662 trajectories.

Layout

  • phase1/<suite>/<task>/task_dag.json: the task DAG; phase1/<suite>/<task>/trajectories/<runId>.json: one label file per run. In these copies the provenance work-directory paths (calls[].space) are relative and dag_sha256 is the sha256 of task_dag.json as published here.
  • Pilot tasks (marked pilot below) were labelled with the same prompts, models and efforts in a 3-task pilot before the full run. pilot_xhigh_dags/ holds the same 3 task DAGs written at effort xhigh, for comparison only (no trajectories were labelled against them).

Tasks

folder trajectories source
phase1/ambench/cabinet_pick_place/ 2 full run
phase1/behavior-1k/assembling_gift_baskets/ 2 full run
phase1/behavior-1k/boxing_books_up_for_storage/ 2 full run
phase1/behavior-1k/boxing_food_after_dinner/ 2 full run
phase1/behavior-1k/bringing_in_wood/ 2 full run
phase1/behavior-1k/bringing_water/ 2 full run
phase1/behavior-1k/can_meat/ 2 full run
phase1/behavior-1k/canning_food/ 2 full run
phase1/behavior-1k/carrying_in_groceries/ 2 full run
phase1/behavior-1k/chopping_wood/ 2 full run
phase1/behavior-1k/clean_a_patio/ 2 full run
phase1/behavior-1k/clean_a_trumpet/ 2 full run
phase1/behavior-1k/clean_up_your_desk/ 2 full run
phase1/behavior-1k/clean_your_rusty_garden_tools/ 2 full run
phase1/behavior-1k/cleaning_up_branches_and_twigs/ 2 full run
phase1/behavior-1k/cleaning_up_plates_and_food/ 2 full run
phase1/behavior-1k/clearing_food_from_table_into_fridge/ 2 full run
phase1/behavior-1k/collecting_aluminum_cans/ 2 full run
phase1/behavior-1k/collecting_childrens_toys/ 2 full run
phase1/behavior-1k/cook_a_brisket/ 2 full run
phase1/behavior-1k/cook_a_frozen_pie/ 2 full run
phase1/behavior-1k/cook_bacon/ 2 pilot
phase1/behavior-1k/cook_broccolini/ 2 full run
phase1/behavior-1k/cook_brussels_sprouts/ 2 full run
phase1/behavior-1k/cook_cabbage/ 2 full run
phase1/behavior-1k/cook_hot_dogs/ 2 full run
phase1/behavior-1k/dispose_of_batteries/ 2 full run
phase1/behavior-1k/freeze_fruit/ 2 full run
phase1/behavior-1k/freeze_pies/ 2 full run
phase1/behavior-1k/laying_tile_floors/ 2 full run
phase1/behavior-1k/loading_the_car/ 2 full run
phase1/behavior-1k/make_gift_bags_for_baby_showers/ 2 full run
phase1/behavior-1k/make_pizza/ 2 full run
phase1/behavior-1k/make_rose_centerpieces/ 2 full run
phase1/behavior-1k/organizing_art_supplies/ 2 full run
phase1/behavior-1k/organizing_school_stuff/ 2 full run
phase1/behavior-1k/outfit_a_basic_toolbox/ 2 full run
phase1/behavior-1k/packing_meal_for_delivery/ 2 full run
phase1/behavior-1k/preparing_lunch_box/ 2 full run
phase1/behavior-1k/put_together_a_basic_pruning_kit/ 2 full run
phase1/behavior-1k/putting_away_toys/ 2 full run
phase1/behavior-1k/putting_dirty_dishes_in_sink/ 2 full run
phase1/behavior-1k/putting_dishes_away_after_cleaning/ 2 full run
phase1/behavior-1k/putting_up_Christmas_decorations_inside/ 2 full run
phase1/behavior-1k/re_shelving_library_books/ 2 full run
phase1/behavior-1k/rearrange_your_room/ 2 full run
phase1/behavior-1k/rearranging_kitchen_furniture/ 2 full run
phase1/behavior-1k/setting_the_fire/ 2 full run
phase1/behavior-1k/setting_the_table/ 2 full run
phase1/behavior-1k/setup_a_bar_for_a_cocktail_party/ 2 full run
phase1/behavior-1k/slicing_vegetables/ 2 full run
phase1/behavior-1k/sorting_books_on_shelf/ 2 full run
phase1/behavior-1k/sorting_bottles_cans_and_paper/ 2 full run
phase1/behavior-1k/sorting_household_items/ 2 full run
phase1/behavior-1k/sorting_vegetables/ 2 full run
phase1/behavior-1k/storing_food/ 2 full run
phase1/behavior-1k/sweeping_garage/ 2 full run
phase1/behavior-1k/tidying_living_room/ 2 full run
phase1/behavior-1k/unloading_the_car/ 2 full run
phase1/behavior-1k/vacuuming_floors/ 2 full run
phase1/behavior-1k/wash_a_baseball_cap/ 2 full run
phase1/behavior-1k/wash_dog_toys/ 2 full run
phase1/bigym2/dishwasher_close/ 2 full run
phase1/bigym2/dishwasher_load_plates/ 2 full run
phase1/bigym2/flip_cutlery/ 2 full run
phase1/bigym2/move_two_plates/ 2 full run
phase1/bigym2/pick_box/ 2 full run
phase1/bigym2/put_cups/ 2 full run
phase1/bigym2/sandwich_remove/ 2 full run
phase1/bigym2/saucepan_to_hob/ 2 full run
phase1/bigym2/wall_cupboard_open/ 2 full run
phase1/dexhand/assemble_hamburger/ 2 full run
phase1/dexhand/batter_whisk/ 2 full run
phase1/dexhand/bicycle_pump_stroke/ 2 full run
phase1/dexhand/binder_page_load/ 2 full run
phase1/dexhand/boot_sole_brush/ 2 full run
phase1/dexhand/drying_rack_unfold/ 2 full run
phase1/dexhand/envelope_letter_insert/ 2 full run
phase1/dexhand/essential_pen_regrasp/ 2 full run
phase1/dexhand/essential_rotary_gait/ 2 full run
phase1/dexhand/fan_guard_install/ 2 full run
phase1/dexhand/picture_hang/ 2 full run
phase1/dexhand/stool_brace_insert/ 2 full run
phase1/dexhand/sweep_dustpan/ 2 full run
phase1/dexhand/usb_plug_insert/ 2 full run
phase1/dexhand/vacuum_bin_reseat/ 2 full run
phase1/dexjoco/bimanual_photograph/ 2 full run
phase1/dextoolbench/brush_blue_brush_sweep_forward/ 2 full run
phase1/dextoolbench/brush_red_brush_sweep_forward/ 2 full run
phase1/dextoolbench/brush_red_brush_sweep_right/ 2 full run
phase1/dextoolbench/eraser_handle_eraser_wipe_c/ 2 full run
phase1/dextoolbench/hammer_claw_hammer_swing_down/ 2 full run
phase1/dextoolbench/hammer_claw_hammer_swing_side/ 2 full run
phase1/dextoolbench/hammer_mallet_hammer_swing_side/ 2 full run
phase1/dextoolbench/marker_sharpie_marker_draw_smile/ 2 full run
phase1/dextoolbench/marker_sharpie_marker_write_c/ 2 full run
phase1/dextoolbench/marker_staples_marker_draw_smile/ 2 full run
phase1/dextoolbench/marker_staples_marker_write_c/ 2 full run
phase1/dextoolbench/screwdriver_long_screwdriver_spin_horizontal/ 2 full run
phase1/dextoolbench/screwdriver_long_screwdriver_spin_vertical/ 2 full run
phase1/dextoolbench/screwdriver_short_screwdriver_spin_vertical/ 2 full run
phase1/dextoolbench/spatula_flat_spatula_flip_over/ 2 full run
phase1/dextoolbench/spatula_flat_spatula_serve_plate/ 2 full run
phase1/dextoolbench/spatula_spoon_spatula_flip_over/ 2 full run
phase1/dextoolbench/spatula_spoon_spatula_serve_plate/ 2 full run
phase1/embodied-olympics-fun/badminton_serve/ 2 full run
phase1/embodied-olympics-fun/balance_board/ 2 full run
phase1/embodied-olympics-fun/box_jump/ 2 full run
phase1/embodied-olympics-fun/clean_and_jerk/ 2 full run
phase1/embodied-olympics-fun/dunk_between_legs/ 2 full run
phase1/embodied-olympics-fun/high_jump/ 2 full run
phase1/embodied-olympics-fun/hurdles_freestanding/ 2 full run
phase1/embodied-olympics-fun/juggling_head/ 2 full run
phase1/embodied-olympics-fun/long_rope/ 2 full run
phase1/embodied-olympics-fun/pole_vault/ 2 full run
phase1/embodied-olympics-fun/standing_long_jump/ 2 full run
phase1/embodied-olympics-fun/steeplechase/ 2 full run
phase1/embodied-olympics-fun/triple_jump/ 2 full run
phase1/embodied-olympics-fun/vault/ 2 full run
phase1/embodied-olympics-fun/volleyball_spike/ 2 pilot
phase1/embodied-olympics-match/badminton_duelrl_i01/ 2 full run
phase1/embodied-olympics-match/badminton_match_i01/ 2 full run
phase1/embodied-olympics-match/badminton_match_i02/ 2 full run
phase1/embodied-olympics-match/badminton_match_i04/ 2 full run
phase1/embodied-olympics-match/badminton_matchlow_i01/ 2 full run
phase1/embodied-olympics-match/fencing_bout_i01/ 2 full run
phase1/embodied-olympics-match/foosball_match_i01/ 2 full run
phase1/embodied-olympics-match/soccer_jointgpu_i01/ 2 full run
phase1/embodied-olympics-match/soccer_match/ 2 full run
phase1/embodied-olympics-match/soccerssl_match/ 2 full run
phase1/embodied-olympics-match/soccerssl_match_i01/ 2 full run
phase1/embodied-olympics-match/tabletennis_duelrl_i01/ 2 full run
phase1/embodied-olympics-match/tabletennis_match_i01/ 2 full run
phase1/embodied-olympics-match/tabletennis_match_i02/ 2 full run
phase1/embodied-olympics-match/tabletennis_match_i03/ 2 full run
phase1/embodied-olympics-match/tabletennis_matchlow_i01/ 2 full run
phase1/embodied-safety-exam/a01_i01/ 2 full run
phase1/embodied-safety-exam/a01_i02/ 2 full run
phase1/embodied-safety-exam/a01_i08/ 2 full run
phase1/embodied-safety-exam/a02_i03/ 2 full run
phase1/embodied-safety-exam/a04/ 2 full run
phase1/embodied-safety-exam/a06_i01/ 2 full run
phase1/embodied-safety-exam/a06_i03/ 2 full run
phase1/embodied-safety-exam/a08_i01/ 2 full run
phase1/embodied-safety-exam/a17/ 2 full run
phase1/embodied-safety-exam/a26_i02/ 2 full run
phase1/embodied-safety-exam/a27_i06/ 2 full run
phase1/embodied-safety-exam/a28/ 2 full run
phase1/embodied-safety-exam/a28_i01/ 2 full run
phase1/embodied-safety-exam/a28_i03/ 2 full run
phase1/embodied-safety-exam/a29_i01/ 2 full run
phase1/embodied-safety-exam/a29_i03/ 2 full run
phase1/embodied-safety-exam/a33_i01/ 2 full run
phase1/embodied-safety-exam/a33_i04/ 2 full run
phase1/embodied-safety-exam/a34_i07/ 2 full run
phase1/embodied-safety-exam/a34_i09/ 2 full run
phase1/embodied-safety-exam/a34_i11/ 2 full run
phase1/embodied-safety-exam/b06_i01/ 2 full run
phase1/embodied-safety-exam/b06_i02/ 2 full run
phase1/embodied-safety-exam/b11/ 2 full run
phase1/embodied-safety-exam/b11_i01/ 2 full run
phase1/embodied-safety-exam/b11_i03/ 2 full run
phase1/embodied-safety-exam/b12_i04/ 2 full run
phase1/embodied-safety-exam/b12_i06/ 2 full run
phase1/embodied-safety-exam/b12_i07/ 2 full run
phase1/embodied-safety-exam/c02/ 2 full run
phase1/embodied-safety-exam/c05_i01/ 2 full run
phase1/embodied-safety-exam/c05_i02/ 2 full run
phase1/embodied-safety-exam/c16/ 2 full run
phase1/embodied-safety-exam/d12_i04/ 2 full run
phase1/embodied-safety-exam/e06/ 2 full run
phase1/embodied-safety-exam/e06_i03/ 2 full run
phase1/embodied-safety-exam/f01/ 2 full run
phase1/embodied-safety-exam/f04_i01/ 2 full run
phase1/embodied-safety-exam/f04_i02/ 2 full run
phase1/embodied-safety-exam/f05/ 2 full run
phase1/embodied-safety-exam/f05_i01/ 2 full run
phase1/embodied-safety-exam/f06/ 2 full run
phase1/embodied-safety-exam/f06_i01/ 2 full run
phase1/embodied-safety-exam/f08/ 2 full run
phase1/embodied-safety-exam/f09/ 2 full run
phase1/embodied-safety-exam/f11/ 2 full run
phase1/humanoidbench/loco_balance_hard/ 2 full run
phase1/humanoidbench/loco_hurdle/ 2 full run
phase1/humanoidbench/manip_bookshelf_simple/ 2 full run
phase1/humanoidbench/manip_door/ 2 full run
phase1/humanoidbench/manip_powerlift/ 2 full run
phase1/humanoidbench/manip_room/ 2 full run
phase1/humanoidbench/manip_window/ 2 full run
phase1/ispatialgym/alignment_axial/ 2 full run
phase1/ispatialgym/camera_relocalization_obstructed/ 2 full run
phase1/ispatialgym/configuration_matching_articulation_open_small_scale/ 2 full run
phase1/ispatialgym/configuration_matching_block_design/ 2 full run
phase1/ispatialgym/inter_object_relation_satisfaction_nextto_small_scale/ 2 full run
phase1/ispatialgym/inter_object_relation_satisfaction_ontop_small_scale/ 2 full run
phase1/ispatialgym/inter_object_relation_satisfaction_touching_small_scale/ 2 full run
phase1/kinder/scoop_pour_i01/ 2 full run
phase1/kinder/sweep_into_drawer/ 2 full run
phase1/kinder/sweep_simple_i01/ 2 full run
phase1/libero-max/goal_wine_bottle_on_wine_rack_top_visual_theme_switch/ 2 full run
phase1/maniguard/cabinet_pickup_i22/ 2 full run
phase1/maniguard/cabinet_pickup_i26/ 2 full run
phase1/maniguard/dusty_transfer_i03/ 2 full run
phase1/maniguard/dusty_transfer_i16/ 2 full run
phase1/maniguard/jar_transport_i13/ 2 full run
phase1/maniguard/lid_transport_i12/ 2 full run
phase1/metaworldplus/button_press_topdown/ 2 full run
phase1/molmospaces/pick/ 2 full run
phase1/mujoco-playground/aloha_single_peg_insertion/ 2 full run
phase1/mujoco-playground/barkour_joystick/ 2 full run
phase1/mujoco-playground/go1_handstand/ 2 full run
phase1/mujoco-playground/go1_joystick_flat_terrain/ 2 full run
phase1/mujoco-playground/spot_flat_terrain_joystick/ 2 full run
phase1/mujoco-playground/spot_joystick_gait_tracking/ 2 full run
phase1/myosuite/baoding/ 2 full run
phase1/myosuite/bimanual/ 2 full run
phase1/myosuite/chase_tag/ 2 full run
phase1/myosuite/dex_airplane_fly/ 2 full run
phase1/myosuite/dex_apple_lift/ 2 full run
phase1/myosuite/dex_bunny_inspect/ 2 full run
phase1/myosuite/dex_cup_pour/ 2 full run
phase1/myosuite/dex_cylinder_pass/ 2 full run
phase1/myosuite/die_reorient/ 2 pilot
phase1/myosuite/obj_reorient/ 2 full run
phase1/myosuite/osl_run/ 2 full run
phase1/myosuite/soccer/ 2 full run
phase1/pandadyn/d01_reciprocal_cup_relay/ 2 full run
phase1/pandadyn/d03_cooperative_puck_course/ 2 full run
phase1/pandadyn/d07_spring_gate_switchyard/ 2 full run
phase1/pandadyn/d09_lever_launch_catch_reload/ 2 full run
phase1/pandadyn/s01_cup_route_delivery/ 2 full run
phase1/pandadyn/s06_cube_flip_route_recover/ 2 full run
phase1/pandadyn/s09_toss_recatch_delivery/ 2 full run
phase1/pandadyn/s11_cylinder_rescue_route_deliver/ 2 full run
phase1/pandadyn/s12_moving_bridge_puck_relay/ 2 full run
phase1/roboarc/blanket_flatten_02/ 2 full run
phase1/roboarc/blanket_flatten_03/ 2 full run
phase1/roboarc/blanket_flatten_07/ 2 full run
phase1/roboarc/blanket_flatten_08/ 2 full run
phase1/roboarc/blanket_fold_01/ 2 full run
phase1/roboarc/blanket_fold_03/ 2 full run
phase1/roboarc/blanket_fold_04/ 2 full run
phase1/roboarc/blanket_fold_05/ 2 full run
phase1/roboarc/buzz_wire_3d_01/ 2 full run
phase1/roboarc/buzz_wire_3d_02/ 2 full run
phase1/roboarc/buzz_wire_3d_04/ 2 full run
phase1/roboarc/buzz_wire_3d_06/ 2 full run
phase1/roboarc/buzz_wire_3d_07/ 2 full run
phase1/roboarc/buzz_wire_3d_09/ 2 full run
phase1/roboarc/buzz_wire_3d_10/ 2 full run
phase1/roboarc/cast_coil_04/ 2 full run
phase1/roboarc/cast_coil_06/ 2 full run
phase1/roboarc/cast_coil_08/ 2 full run
phase1/roboarc/cast_keyhole_03/ 2 full run
phase1/roboarc/cast_keyhole_04/ 2 full run
phase1/roboarc/cast_keyhole_05/ 2 full run
phase1/roboarc/fiber_bundling_04/ 2 full run
phase1/roboarc/fiber_bundling_06/ 2 full run
phase1/roboarc/fiber_bundling_07/ 2 full run
phase1/roboarc/fiber_bundling_09/ 2 full run
phase1/roboarc/fiber_bundling_10/ 2 full run
phase1/roboarc/hedgehog_in_cage_05/ 2 full run
phase1/roboarc/hedgehog_in_cage_06/ 2 full run
phase1/roboarc/hedgehog_in_cage_07/ 2 full run
phase1/roboarc/horseshoe_ring_02/ 2 full run
phase1/roboarc/horseshoe_ring_03/ 2 full run
phase1/roboarc/interlocked_claws_05/ 2 full run
phase1/roboarc/interlocked_claws_09/ 2 full run
phase1/roboarc/interlocked_claws_11/ 2 full run
phase1/roboarc/interlocked_claws_12/ 2 full run
phase1/roboarc/knot_untying_06/ 2 full run
phase1/roboarc/knot_untying_09/ 2 full run
phase1/roboarc/lock_picking_04/ 2 full run
phase1/roboarc/lock_picking_05/ 2 full run
phase1/roboarc/rope_threading_03/ 2 full run
phase1/roboarc/rope_threading_04/ 2 full run
phase1/roboarc/rope_threading_05/ 2 full run
phase1/roboarc/rope_threading_06/ 2 full run
phase1/roboarc/rope_threading_07/ 2 full run
phase1/roboarc/rope_threading_08/ 2 full run
phase1/roboarc/slotted_post_ring_04/ 2 full run
phase1/roboarc/slotted_post_ring_05/ 2 full run
phase1/robocasa/turn_off_sink_faucet/ 2 full run
phase1/robocasa365/assemble_cooking_array/ 2 full run
phase1/robocasa365/cereal_and_bowl/ 2 full run
phase1/robocasa365/microwave_defrost_meat/ 2 full run
phase1/robocasa365/searing_meat/ 2 full run
phase1/robodojo/fill_egg_holder/ 2 full run
phase1/robodojo/hang_mugs/ 2 full run
phase1/robodojo/insert_key/ 2 full run
phase1/robodojo/make_kong/ 2 full run
phase1/robodojo/make_toast/ 2 full run
phase1/robodojo/pack_objects_into_box/ 2 full run
phase1/robodojo/store_laptop_and_headphones/ 2 full run
phase1/robodojo/store_laptop_and_headphones_random/ 2 full run
phase1/robodojo/store_tools_in_toolbox/ 2 full run
phase1/robodojo/swap_blocks/ 2 full run
phase1/robodojo/sweep_blocks_random/ 2 full run
phase1/robolab/bbq_sauce_in_bin/ 2 full run
phase1/robolab/food_packing3_cans/ 2 full run
phase1/robolab/put_two_mugs_on_shelf/ 2 full run
phase1/robolab/utensils_in_mug/ 2 full run
phase1/robolab/white_mug_in_center_of_table/ 2 full run
phase1/robopaint/acrylic_applesoranges/ 2 full run
phase1/robopaint/acrylic_bedroom/ 2 full run
phase1/robopaint/acrylic_bluehorses/ 2 full run
phase1/robopaint/acrylic_greatwave/ 2 full run
phase1/robopaint/acrylic_plumpark/ 2 full run
phase1/robopaint/acrylic_redfuji/ 2 full run
phase1/robopaint/acrylic_suddenshower/ 2 full run
phase1/robopaint/acrylic_sunflowers/ 2 full run
phase1/robopaint/acrylic_vision/ 2 full run
phase1/robopaint/caoshu_beiquelindanshui/ 2 full run
phase1/robopaint/caoshu_siwushaonian/ 2 full run
phase1/robopaint/caoshu_zhen/ 2 full run
phase1/robopaint/oil_monalisa_i01/ 2 full run
phase1/robopaint/oil_starrynight_i01/ 2 full run
phase1/robopaint/xingshu_feng/ 2 full run
phase1/robopaint/xingshu_he/ 2 full run
phase1/robopaint/xingshu_huifenghechang/ 2 full run
phase1/robopaint/xingshu_maolinxiuzhu/ 2 full run
phase1/robopaint/xingshu_xian/ 2 full run
phase1/robopaint/xingshu_yong/ 2 full run
phase1/robotwin-2/hanging_mug/ 2 full run
phase1/robotwin-2/place_bread_skillet/ 2 full run
phase1/robowits/cover_with_lid/ 2 full run
phase1/robowits/stack_bowls/ 2 full run
phase1/robowits/stand_bulb/ 2 full run
phase1/robowits/stand_pages/ 2 full run
phase1/vlabench/texas_holdem/ 2 full run
phase1/yamlab/put_pot_on_cooktop/ 2 full run

Phase 2

  • Prompts: v3.2 (the frozen labelling prompts with the fixes from a blind recheck of 30 phase-1 tasks; taxonomy v2; schemas efe-task-dag/1 and efe-trajectory-labels/1).
  • Writer: Codex gpt-6.1-sol (effort high for task DAGs and trajectories). Reviewer: Claude claude-sonnet-5-5; up to two fix rounds plus a confirm round; files still disputed after that carry review.status: "disputed".
  • Task repository: embodied-first-exam commit 80cac7a5a227db930c659c3307a1e79c2a84bc53.
  • Scope: every benchmark task except the roboengineer suite (434 tasks), two runs per task (one success when there is one, plus a failure; runs with a video first; seeded pick), 868 runs. Every task gets a newly written DAG, and all its phase-2 runs (including runs already labelled in phase 1) are labelled against that DAG.
  • Remaining runs (added from 2026-10-29): every other run of each in-scope task (6,648 runs in all) is labelled against the task's phase-2 DAG as published here (the DAG is not rewritten), with the same prompts, models and efforts, and added to the same trajectories/ folder; the run that served as a task's reference success (oracle) when its DAG was written stays that task's oracle. Runs without a graded video are labelled from the log only. Status: 5,780 of the 5,780 remaining runs labelled and uploaded (complete).
  • Run catalog: built from the run index of the dataset MLL-Lab/embodied-exam-runs (the run viewer's run.json files); videos as in phase 1 from MLL-Lab/embodied-exam-data, runs/<runId>/replay.mp4. Some picked runs and tasks are newer than the catalog shipped with the labelling interface, so importing them needs a catalog refreshed from that run index.
  • The importer prefers the highest phase: when a task is under phase1/ and phase2/, only phase2/<suite>/<task>/ (its DAG and its own trajectories) is used; phase1/ stays as a record.
  • Layout as in phase 1: phase2/<suite>/<task>/task_dag.json and phase2/<suite>/<task>/trajectories/<runId>.json, provenance work-directory paths relative, dag_sha256 = sha256 of task_dag.json as published here. A task is uploaded whole (its DAG together with all its phase-2 trajectories); remaining-run trajectories are added batch by batch once the batch passes validation.
  • Phase 2 complete: every task of the run passed validation and is listed below. Status: 434 tasks, 868 trajectories. With the remaining runs: 6,648 trajectories in all.

Phase 2 tasks

folder trajectories also in phase 1
phase2/ambench/cabinet_pick_place/ 15 yes
phase2/ambench/peg_in_hole/ 15 no
phase2/behavior-1k/assembling_gift_baskets/ 16 yes
phase2/behavior-1k/boxing_books_up_for_storage/ 16 yes
phase2/behavior-1k/boxing_food_after_dinner/ 15 yes
phase2/behavior-1k/bringing_in_wood/ 16 yes
phase2/behavior-1k/bringing_paper_to_recycling/ 16 no
phase2/behavior-1k/bringing_water/ 16 yes
phase2/behavior-1k/can_meat/ 15 yes
phase2/behavior-1k/canning_food/ 15 yes
phase2/behavior-1k/carrying_in_groceries/ 16 yes
phase2/behavior-1k/carrying_out_garden_furniture/ 16 no
phase2/behavior-1k/chopping_wood/ 16 yes
phase2/behavior-1k/clean_a_patio/ 16 yes
phase2/behavior-1k/clean_a_trumpet/ 15 yes
phase2/behavior-1k/clean_boxing_gloves/ 16 no
phase2/behavior-1k/clean_up_your_desk/ 16 yes
phase2/behavior-1k/clean_your_rusty_garden_tools/ 16 yes
phase2/behavior-1k/cleaning_up_branches_and_twigs/ 16 yes
phase2/behavior-1k/cleaning_up_plates_and_food/ 14 yes
phase2/behavior-1k/clearing_food_from_table_into_fridge/ 15 yes
phase2/behavior-1k/collecting_aluminum_cans/ 16 yes
phase2/behavior-1k/collecting_childrens_toys/ 16 yes
phase2/behavior-1k/cook_a_brisket/ 16 yes
phase2/behavior-1k/cook_a_frozen_pie/ 15 yes
phase2/behavior-1k/cook_bacon/ 16 yes
phase2/behavior-1k/cook_broccolini/ 16 yes
phase2/behavior-1k/cook_brussels_sprouts/ 15 yes
phase2/behavior-1k/cook_cabbage/ 16 yes
phase2/behavior-1k/cook_hot_dogs/ 14 yes
phase2/behavior-1k/dispose_of_batteries/ 16 yes
phase2/behavior-1k/freeze_fruit/ 14 yes
phase2/behavior-1k/freeze_pies/ 16 yes
phase2/behavior-1k/halve_an_egg/ 15 no
phase2/behavior-1k/installing_a_fax_machine/ 15 no
phase2/behavior-1k/laying_tile_floors/ 15 yes
phase2/behavior-1k/loading_the_car/ 14 yes
phase2/behavior-1k/make_gift_bags_for_baby_showers/ 16 yes
phase2/behavior-1k/make_pizza/ 15 yes
phase2/behavior-1k/make_rose_centerpieces/ 14 yes
phase2/behavior-1k/organizing_art_supplies/ 13 yes
phase2/behavior-1k/organizing_school_stuff/ 16 yes
phase2/behavior-1k/outfit_a_basic_toolbox/ 14 yes
phase2/behavior-1k/packing_meal_for_delivery/ 14 yes
phase2/behavior-1k/polishing_shoes/ 13 no
phase2/behavior-1k/preparing_lunch_box/ 13 yes
phase2/behavior-1k/put_together_a_basic_pruning_kit/ 16 yes
phase2/behavior-1k/putting_away_toys/ 15 yes
phase2/behavior-1k/putting_dirty_dishes_in_sink/ 14 yes
phase2/behavior-1k/putting_dishes_away_after_cleaning/ 16 yes
phase2/behavior-1k/putting_up_Christmas_decorations_inside/ 16 yes
phase2/behavior-1k/re_shelving_library_books/ 16 yes
phase2/behavior-1k/rearrange_your_room/ 15 yes
phase2/behavior-1k/rearranging_kitchen_furniture/ 12 yes
phase2/behavior-1k/setting_the_fire/ 16 yes
phase2/behavior-1k/setting_the_table/ 15 yes
phase2/behavior-1k/setup_a_bar_for_a_cocktail_party/ 16 yes
phase2/behavior-1k/slicing_vegetables/ 15 yes
phase2/behavior-1k/sorting_books_on_shelf/ 14 yes
phase2/behavior-1k/sorting_bottles_cans_and_paper/ 14 yes
phase2/behavior-1k/sorting_household_items/ 15 yes
phase2/behavior-1k/sorting_vegetables/ 15 yes
phase2/behavior-1k/store_batteries/ 14 no
phase2/behavior-1k/store_produce/ 15 no
phase2/behavior-1k/storing_food/ 15 yes
phase2/behavior-1k/sweeping_garage/ 14 yes
phase2/behavior-1k/thawing_frozen_food/ 15 no
phase2/behavior-1k/tidying_living_room/ 14 yes
phase2/behavior-1k/turning_out_all_lights_before_sleep/ 15 no
phase2/behavior-1k/unloading_the_car/ 15 yes
phase2/behavior-1k/vacuuming_floors/ 15 yes
phase2/behavior-1k/wash_a_baseball_cap/ 14 yes
phase2/behavior-1k/wash_dog_toys/ 14 yes
phase2/bigym2/dishwasher_close/ 22 yes
phase2/bigym2/dishwasher_load_plates/ 24 yes
phase2/bigym2/flip_cutlery/ 24 yes
phase2/bigym2/move_two_plates/ 24 yes
phase2/bigym2/pick_box/ 24 yes
phase2/bigym2/put_cups/ 24 yes
phase2/bigym2/sandwich_remove/ 24 yes
phase2/bigym2/saucepan_to_hob/ 24 yes
phase2/bigym2/stack_blocks/ 24 no
phase2/bigym2/wall_cupboard_open/ 24 yes
phase2/dexhand/assemble_hamburger/ 16 yes
phase2/dexhand/batter_whisk/ 16 yes
phase2/dexhand/bicycle_pump_stroke/ 16 yes
phase2/dexhand/binder_page_load/ 16 yes
phase2/dexhand/book_page_turn/ 16 no
phase2/dexhand/boot_sole_brush/ 16 yes
phase2/dexhand/drying_rack_unfold/ 16 yes
phase2/dexhand/envelope_letter_insert/ 16 yes
phase2/dexhand/essential_cube_reorient/ 16 no
phase2/dexhand/essential_pen_regrasp/ 16 yes
phase2/dexhand/essential_rotary_gait/ 10 yes
phase2/dexhand/essential_two_object/ 16 no
phase2/dexhand/fan_guard_install/ 16 yes
phase2/dexhand/grinder_crank/ 16 no
phase2/dexhand/jar_lid_open/ 15 no
phase2/dexhand/pancake_serve/ 16 no
phase2/dexhand/picture_hang/ 16 yes
phase2/dexhand/ram_module_install/ 16 no
phase2/dexhand/remote_battery_swap/ 16 no
phase2/dexhand/stool_brace_insert/ 16 yes
phase2/dexhand/sweep_dustpan/ 16 yes
phase2/dexhand/usb_plug_insert/ 16 yes
phase2/dexhand/vacuum_bin_reseat/ 16 yes
phase2/dexjoco/bimanual_photograph/ 16 yes
phase2/dexjoco/fold_glasses/ 16 no
phase2/dextoolbench/brush_blue_brush_sweep_forward/ 16 yes
phase2/dextoolbench/brush_red_brush_sweep_forward/ 16 yes
phase2/dextoolbench/brush_red_brush_sweep_right/ 16 yes
phase2/dextoolbench/eraser_handle_eraser_wipe_c/ 15 yes
phase2/dextoolbench/hammer_claw_hammer_swing_down/ 16 yes
phase2/dextoolbench/hammer_claw_hammer_swing_side/ 16 yes
phase2/dextoolbench/hammer_mallet_hammer_swing_down/ 16 no
phase2/dextoolbench/hammer_mallet_hammer_swing_side/ 15 yes
phase2/dextoolbench/marker_sharpie_marker_draw_smile/ 16 yes
phase2/dextoolbench/marker_sharpie_marker_write_c/ 16 yes
phase2/dextoolbench/marker_staples_marker_draw_smile/ 16 yes
phase2/dextoolbench/marker_staples_marker_write_c/ 16 yes
phase2/dextoolbench/screwdriver_long_screwdriver_spin_horizontal/ 16 yes
phase2/dextoolbench/screwdriver_long_screwdriver_spin_vertical/ 16 yes
phase2/dextoolbench/screwdriver_short_screwdriver_spin_horizontal/ 16 no
phase2/dextoolbench/screwdriver_short_screwdriver_spin_vertical/ 16 yes
phase2/dextoolbench/spatula_flat_spatula_flip_over/ 16 yes
phase2/dextoolbench/spatula_flat_spatula_serve_plate/ 16 yes
phase2/dextoolbench/spatula_spoon_spatula_flip_over/ 16 yes
phase2/dextoolbench/spatula_spoon_spatula_serve_plate/ 16 yes
phase2/embodied-olympics-fun/alpine_slalom/ 16 no
phase2/embodied-olympics-fun/backflip/ 16 no
phase2/embodied-olympics-fun/badminton_serve/ 16 yes
phase2/embodied-olympics-fun/balance_beam/ 15 no
phase2/embodied-olympics-fun/balance_board/ 16 yes
phase2/embodied-olympics-fun/basketball_dribble/ 16 no
phase2/embodied-olympics-fun/bowling/ 16 no
phase2/embodied-olympics-fun/box_jump/ 15 yes
phase2/embodied-olympics-fun/boxing/ 16 no
phase2/embodied-olympics-fun/catch/ 24 no
phase2/embodied-olympics-fun/catch_and_shoot/ 24 no
phase2/embodied-olympics-fun/clean_and_jerk/ 16 yes
phase2/embodied-olympics-fun/discus/ 16 no
phase2/embodied-olympics-fun/dodgeball/ 24 no
phase2/embodied-olympics-fun/dribble_slalom/ 16 no
phase2/embodied-olympics-fun/dunk/ 16 no
phase2/embodied-olympics-fun/dunk_between_legs/ 16 yes
phase2/embodied-olympics-fun/free_kick_wall/ 15 no
phase2/embodied-olympics-fun/free_throw/ 16 no
phase2/embodied-olympics-fun/freeski_360/ 16 no
phase2/embodied-olympics-fun/goalkeeper_save/ 24 no
phase2/embodied-olympics-fun/golf_putt/ 16 no
phase2/embodied-olympics-fun/high_bar_swing/ 16 no
phase2/embodied-olympics-fun/high_jump/ 16 yes
phase2/embodied-olympics-fun/hurdles_freestanding/ 16 yes
phase2/embodied-olympics-fun/javelin/ 16 no
phase2/embodied-olympics-fun/juggle_three/ 24 no
phase2/embodied-olympics-fun/juggling/ 16 no
phase2/embodied-olympics-fun/juggling_feet/ 16 no
phase2/embodied-olympics-fun/juggling_head/ 24 yes
phase2/embodied-olympics-fun/jump_rope/ 16 no
phase2/embodied-olympics-fun/layup/ 16 no
phase2/embodied-olympics-fun/long_jump/ 16 no
phase2/embodied-olympics-fun/long_rope/ 23 yes
phase2/embodied-olympics-fun/paddle_bounce/ 24 no
phase2/embodied-olympics-fun/pitch_target/ 16 no
phase2/embodied-olympics-fun/platform_10m/ 14 no
phase2/embodied-olympics-fun/pole_vault/ 16 yes
phase2/embodied-olympics-fun/rugby_conversion/ 16 no
phase2/embodied-olympics-fun/shot_put/ 15 no
phase2/embodied-olympics-fun/slalom/ 16 no
phase2/embodied-olympics-fun/springboard_3m/ 16 no
phase2/embodied-olympics-fun/sprint/ 16 no
phase2/embodied-olympics-fun/standing_long_jump/ 15 yes
phase2/embodied-olympics-fun/steeplechase/ 16 yes
phase2/embodied-olympics-fun/table_tennis_return/ 23 no
phase2/embodied-olympics-fun/taekwondo_kick/ 15 no
phase2/embodied-olympics-fun/tennis_return/ 23 no
phase2/embodied-olympics-fun/tennis_serve/ 16 no
phase2/embodied-olympics-fun/toes_to_bar/ 16 no
phase2/embodied-olympics-fun/trampoline/ 16 no
phase2/embodied-olympics-fun/triple_jump/ 16 yes
phase2/embodied-olympics-fun/vault/ 16 yes
phase2/embodied-olympics-fun/volleyball_serve/ 16 no
phase2/embodied-olympics-fun/volleyball_spike/ 24 yes
phase2/embodied-olympics-fun/wall_ball/ 16 no
phase2/embodied-olympics-match/badminton_duelrl_i01/ 8 yes
phase2/embodied-olympics-match/badminton_match_i01/ 16 yes
phase2/embodied-olympics-match/badminton_match_i02/ 16 yes
phase2/embodied-olympics-match/badminton_match_i04/ 8 yes
phase2/embodied-olympics-match/badminton_matchlow_i01/ 7 yes
phase2/embodied-olympics-match/fencing_bout_i01/ 7 yes
phase2/embodied-olympics-match/foosball_match_i01/ 8 yes
phase2/embodied-olympics-match/soccer_jointcpu_i01/ 8 no
phase2/embodied-olympics-match/soccer_jointgpu_i01/ 7 yes
phase2/embodied-olympics-match/soccer_match/ 16 yes
phase2/embodied-olympics-match/soccerssl_match/ 15 yes
phase2/embodied-olympics-match/soccerssl_match_i01/ 8 yes
phase2/embodied-olympics-match/tabletennis_duelrl_i01/ 8 yes
phase2/embodied-olympics-match/tabletennis_match_i01/ 16 yes
phase2/embodied-olympics-match/tabletennis_match_i02/ 16 yes
phase2/embodied-olympics-match/tabletennis_match_i03/ 8 yes
phase2/embodied-olympics-match/tabletennis_match_i04/ 8 no
phase2/embodied-olympics-match/tabletennis_matchlow_i01/ 7 yes
phase2/embodied-safety-exam/a01_i01/ 8 yes
phase2/embodied-safety-exam/a01_i02/ 8 yes
phase2/embodied-safety-exam/a01_i04/ 8 no
phase2/embodied-safety-exam/a01_i08/ 8 yes
phase2/embodied-safety-exam/a02_i03/ 8 yes
phase2/embodied-safety-exam/a04/ 8 yes
phase2/embodied-safety-exam/a06_i01/ 8 yes
phase2/embodied-safety-exam/a06_i03/ 8 yes
phase2/embodied-safety-exam/a08_i01/ 8 yes
phase2/embodied-safety-exam/a17/ 8 yes
phase2/embodied-safety-exam/a26_i02/ 8 yes
phase2/embodied-safety-exam/a27_i06/ 8 yes
phase2/embodied-safety-exam/a28/ 8 yes
phase2/embodied-safety-exam/a28_i01/ 8 yes
phase2/embodied-safety-exam/a28_i03/ 8 yes
phase2/embodied-safety-exam/a29/ 8 no
phase2/embodied-safety-exam/a29_i01/ 8 yes
phase2/embodied-safety-exam/a29_i03/ 8 yes
phase2/embodied-safety-exam/a33_i01/ 8 yes
phase2/embodied-safety-exam/a33_i04/ 8 yes
phase2/embodied-safety-exam/a34_i07/ 8 yes
phase2/embodied-safety-exam/a34_i09/ 8 yes
phase2/embodied-safety-exam/a34_i11/ 8 yes
phase2/embodied-safety-exam/b06_i01/ 8 yes
phase2/embodied-safety-exam/b06_i02/ 8 yes
phase2/embodied-safety-exam/b11/ 8 yes
phase2/embodied-safety-exam/b11_i01/ 7 yes
phase2/embodied-safety-exam/b11_i03/ 8 yes
phase2/embodied-safety-exam/b12_i04/ 8 yes
phase2/embodied-safety-exam/b12_i06/ 8 yes
phase2/embodied-safety-exam/b12_i07/ 8 yes
phase2/embodied-safety-exam/c02/ 16 yes
phase2/embodied-safety-exam/c02_i01/ 16 no
phase2/embodied-safety-exam/c04_i01/ 15 no
phase2/embodied-safety-exam/c05_i01/ 16 yes
phase2/embodied-safety-exam/c05_i02/ 8 yes
phase2/embodied-safety-exam/c16/ 8 yes
phase2/embodied-safety-exam/d12_i04/ 8 yes
phase2/embodied-safety-exam/e06/ 8 yes
phase2/embodied-safety-exam/e06_i03/ 8 yes
phase2/embodied-safety-exam/f01/ 8 yes
phase2/embodied-safety-exam/f04_i01/ 8 yes
phase2/embodied-safety-exam/f04_i02/ 8 yes
phase2/embodied-safety-exam/f05/ 8 yes
phase2/embodied-safety-exam/f05_i01/ 8 yes
phase2/embodied-safety-exam/f06/ 8 yes
phase2/embodied-safety-exam/f06_i01/ 8 yes
phase2/embodied-safety-exam/f08/ 8 yes
phase2/embodied-safety-exam/f09/ 8 yes
phase2/embodied-safety-exam/f11/ 8 yes
phase2/humanoidbench/loco_balance_hard/ 23 yes
phase2/humanoidbench/loco_hurdle/ 24 yes
phase2/humanoidbench/manip_bookshelf_simple/ 24 yes
phase2/humanoidbench/manip_cube/ 24 no
phase2/humanoidbench/manip_door/ 23 yes
phase2/humanoidbench/manip_powerlift/ 24 yes
phase2/humanoidbench/manip_room/ 24 yes
phase2/humanoidbench/manip_window/ 24 yes
phase2/ispatialgym/alignment_axial/ 16 yes
phase2/ispatialgym/camera_relocalization_obstructed/ 16 yes
phase2/ispatialgym/configuration_matching_articulation_open_small_scale/ 16 yes
phase2/ispatialgym/configuration_matching_block_design/ 15 yes
phase2/ispatialgym/inter_object_relation_satisfaction_nextto_small_scale/ 15 yes
phase2/ispatialgym/inter_object_relation_satisfaction_ontop_small_scale/ 16 yes
phase2/ispatialgym/inter_object_relation_satisfaction_touching_small_scale/ 16 yes
phase2/ispatialgym/mental_rotation/ 15 no
phase2/kinder/constrained_cupboard/ 16 no
phase2/kinder/scoop_pour_i01/ 16 yes
phase2/kinder/sweep_into_drawer/ 16 yes
phase2/kinder/sweep_simple_i01/ 16 yes
phase2/libero-max/goal_akita_black_bowl_in_wooden_cabinet_top_target_relocation/ 16 no
phase2/libero-max/goal_wine_bottle_on_wine_rack_top_visual_theme_switch/ 16 yes
phase2/maniguard/cabinet_pickup_i22/ 16 yes
phase2/maniguard/cabinet_pickup_i26/ 16 yes
phase2/maniguard/dusty_transfer_i03/ 16 yes
phase2/maniguard/dusty_transfer_i16/ 16 yes
phase2/maniguard/jar_transport_i13/ 16 yes
phase2/maniguard/jar_transport_i25/ 16 no
phase2/maniguard/lid_transport_i12/ 16 yes
phase2/metaworldplus/basketball/ 16 no
phase2/metaworldplus/button_press_topdown/ 15 yes
phase2/molmospaces/navigate_to/ 16 no
phase2/molmospaces/pick/ 16 yes
phase2/mujoco-playground/aloha_single_peg_insertion/ 16 yes
phase2/mujoco-playground/barkour_joystick/ 24 yes
phase2/mujoco-playground/go1_handstand/ 16 yes
phase2/mujoco-playground/go1_joystick_flat_terrain/ 16 yes
phase2/mujoco-playground/go1_joystick_rough_terrain/ 16 no
phase2/mujoco-playground/spot_flat_terrain_joystick/ 16 yes
phase2/mujoco-playground/spot_joystick_gait_tracking/ 16 yes
phase2/myosuite/baoding/ 16 yes
phase2/myosuite/bimanual/ 16 yes
phase2/myosuite/chase_tag/ 16 yes
phase2/myosuite/dex_airplane_fly/ 16 yes
phase2/myosuite/dex_apple_lift/ 16 yes
phase2/myosuite/dex_bunny_inspect/ 16 yes
phase2/myosuite/dex_cup_pour/ 16 yes
phase2/myosuite/dex_cylinder_pass/ 16 yes
phase2/myosuite/dex_hammer_use/ 16 no
phase2/myosuite/die_reorient/ 16 yes
phase2/myosuite/obj_reorient/ 16 yes
phase2/myosuite/osl_run/ 16 yes
phase2/myosuite/soccer/ 16 yes
phase2/pandadyn/d01_reciprocal_cup_relay/ 24 yes
phase2/pandadyn/d03_cooperative_puck_course/ 23 yes
phase2/pandadyn/d05_moving_ramp_relay/ 16 no
phase2/pandadyn/d07_spring_gate_switchyard/ 24 yes
phase2/pandadyn/d09_lever_launch_catch_reload/ 24 yes
phase2/pandadyn/s01_cup_route_delivery/ 24 yes
phase2/pandadyn/s06_cube_flip_route_recover/ 16 yes
phase2/pandadyn/s09_toss_recatch_delivery/ 23 yes
phase2/pandadyn/s11_cylinder_rescue_route_deliver/ 24 yes
phase2/pandadyn/s12_moving_bridge_puck_relay/ 24 yes
phase2/roboarc/blanket_flatten_02/ 14 yes
phase2/roboarc/blanket_flatten_03/ 15 yes
phase2/roboarc/blanket_flatten_04/ 15 no
phase2/roboarc/blanket_flatten_07/ 15 yes
phase2/roboarc/blanket_flatten_08/ 15 yes
phase2/roboarc/blanket_fold_01/ 15 yes
phase2/roboarc/blanket_fold_03/ 15 yes
phase2/roboarc/blanket_fold_04/ 15 yes
phase2/roboarc/blanket_fold_05/ 16 yes
phase2/roboarc/buzz_wire_3d_01/ 16 yes
phase2/roboarc/buzz_wire_3d_02/ 16 yes
phase2/roboarc/buzz_wire_3d_04/ 16 yes
phase2/roboarc/buzz_wire_3d_06/ 16 yes
phase2/roboarc/buzz_wire_3d_07/ 16 yes
phase2/roboarc/buzz_wire_3d_09/ 16 yes
phase2/roboarc/buzz_wire_3d_10/ 16 yes
phase2/roboarc/cast_coil_04/ 15 yes
phase2/roboarc/cast_coil_06/ 16 yes
phase2/roboarc/cast_coil_07/ 16 no
phase2/roboarc/cast_coil_08/ 16 yes
phase2/roboarc/cast_keyhole_03/ 16 yes
phase2/roboarc/cast_keyhole_04/ 16 yes
phase2/roboarc/cast_keyhole_05/ 16 yes
phase2/roboarc/fiber_bundling_04/ 16 yes
phase2/roboarc/fiber_bundling_06/ 16 yes
phase2/roboarc/fiber_bundling_07/ 16 yes
phase2/roboarc/fiber_bundling_09/ 16 yes
phase2/roboarc/fiber_bundling_10/ 16 yes
phase2/roboarc/hedgehog_in_cage_05/ 16 yes
phase2/roboarc/hedgehog_in_cage_06/ 16 yes
phase2/roboarc/hedgehog_in_cage_07/ 16 yes
phase2/roboarc/horseshoe_ring_02/ 16 yes
phase2/roboarc/horseshoe_ring_03/ 16 yes
phase2/roboarc/ikea_eket/ 16 no
phase2/roboarc/ikea_gladom/ 16 no
phase2/roboarc/ikea_ikea365/ 16 no
phase2/roboarc/ikea_lack/ 16 no
phase2/roboarc/ikea_stegesund/ 16 no
phase2/roboarc/ikea_svallet/ 16 no
phase2/roboarc/ikea_variera/ 16 no
phase2/roboarc/ikea_vesken/ 16 no
phase2/roboarc/interlocked_claws_05/ 16 yes
phase2/roboarc/interlocked_claws_09/ 15 yes
phase2/roboarc/interlocked_claws_11/ 16 yes
phase2/roboarc/interlocked_claws_12/ 16 yes
phase2/roboarc/knot_untying_06/ 16 yes
phase2/roboarc/knot_untying_09/ 15 yes
phase2/roboarc/lock_picking_04/ 16 yes
phase2/roboarc/lock_picking_05/ 16 yes
phase2/roboarc/rope_threading_03/ 15 yes
phase2/roboarc/rope_threading_04/ 16 yes
phase2/roboarc/rope_threading_05/ 15 yes
phase2/roboarc/rope_threading_06/ 15 yes
phase2/roboarc/rope_threading_07/ 16 yes
phase2/roboarc/rope_threading_08/ 15 yes
phase2/roboarc/slotted_post_ring_03/ 15 no
phase2/roboarc/slotted_post_ring_04/ 16 yes
phase2/roboarc/slotted_post_ring_05/ 16 yes
phase2/robocasa-gr1/potato_to_microwave_close/ 16 no
phase2/robocasa/coffee_setup_mug/ 16 no
phase2/robocasa/turn_off_sink_faucet/ 16 yes
phase2/robocasa365/assemble_cooking_array/ 16 yes
phase2/robocasa365/cereal_and_bowl/ 16 yes
phase2/robocasa365/display_meat_variety/ 16 no
phase2/robocasa365/microwave_defrost_meat/ 16 yes
phase2/robocasa365/searing_meat/ 16 yes
phase2/robodojo/fill_egg_holder/ 16 yes
phase2/robodojo/hang_mugs/ 16 yes
phase2/robodojo/hang_mugs_random/ 15 no
phase2/robodojo/insert_key/ 16 yes
phase2/robodojo/make_kong/ 16 yes
phase2/robodojo/make_toast/ 16 yes
phase2/robodojo/pack_objects_into_box/ 16 yes
phase2/robodojo/store_laptop_and_headphones/ 15 yes
phase2/robodojo/store_laptop_and_headphones_random/ 16 yes
phase2/robodojo/store_tools_in_toolbox/ 16 yes
phase2/robodojo/swap_blocks/ 16 yes
phase2/robodojo/sweep_blocks_random/ 16 yes
phase2/robolab/bbq_sauce_in_bin/ 14 yes
phase2/robolab/food_packing3_cans/ 14 yes
phase2/robolab/put_two_mugs_on_shelf/ 15 yes
phase2/robolab/red_dishes_in_bin/ 13 no
phase2/robolab/utensils_in_mug/ 15 yes
phase2/robolab/white_mug_in_center_of_table/ 16 yes
phase2/robopaint/acrylic_applesoranges/ 16 yes
phase2/robopaint/acrylic_bedroom/ 16 yes
phase2/robopaint/acrylic_bluehorses/ 16 yes
phase2/robopaint/acrylic_greatwave/ 15 yes
phase2/robopaint/acrylic_kajikazawa/ 16 no
phase2/robopaint/acrylic_plumpark/ 16 yes
phase2/robopaint/acrylic_redfuji/ 16 yes
phase2/robopaint/acrylic_suddenshower/ 16 yes
phase2/robopaint/acrylic_sunflowers/ 15 yes
phase2/robopaint/acrylic_vision/ 16 yes
phase2/robopaint/caoshu_beiquelindanshui/ 16 yes
phase2/robopaint/caoshu_siwushaonian/ 16 yes
phase2/robopaint/caoshu_zhen/ 16 yes
phase2/robopaint/kaishu_linfengguilong/ 16 no
phase2/robopaint/oil_monalisa_i01/ 16 yes
phase2/robopaint/oil_starrynight_i01/ 16 yes
phase2/robopaint/xingshu_feng/ 16 yes
phase2/robopaint/xingshu_he/ 16 yes
phase2/robopaint/xingshu_huifenghechang/ 16 yes
phase2/robopaint/xingshu_maolinxiuzhu/ 16 yes
phase2/robopaint/xingshu_xian/ 16 yes
phase2/robopaint/xingshu_yong/ 16 yes
phase2/robotwin-2/hanging_mug/ 16 yes
phase2/robotwin-2/place_bread_skillet/ 16 yes
phase2/robotwin-2/put_bottles_dustbin/ 16 no
phase2/robowits/ball_into_bottle/ 16 no
phase2/robowits/cover_with_lid/ 16 yes
phase2/robowits/retrieve_cube/ 16 no
phase2/robowits/stack_bowls/ 15 yes
phase2/robowits/stand_bulb/ 16 yes
phase2/robowits/stand_pages/ 16 yes
phase2/vlabench/heat_food/ 16 no
phase2/vlabench/texas_holdem/ 16 yes
phase2/yamlab/hang_mug_on_tree_i15/ 16 no
phase2/yamlab/put_pot_on_cooktop/ 16 yes

Phase K: Kimi 3.1 Pro (2026-11-08)

  • Runs: the counted attempt of Claude Code with Kimi 3.1 Pro (run claude_code-kimi_3_1_pro-high) on each core test (privileged, standard and realtime), successes and failures alike, the same rule as for the other seven agents. Tests whose task has no phase-2 DAG are not labelled (no DAG is written here): embodiedswe, roboengineer and two newer Embodied Olympics - Match tasks.
  • Writer: Claude Sonnet 5.5 (claude-sonnet-5-5, Claude Code) for this agent only, by the owner's decision of 2026-11-08 (no GPT-6.1 Sol quota). The reviewer is Claude Sonnet 5.5 as before, in a separate session. Everything else is unchanged: prompts v3.2, the task's phase-2 DAG as published under phase2/ (not rewritten; dag_sha256 is that file), the same task inputs and oracle rule, the same review rounds and validation. Each file's provenance says engine: claude, model: claude-sonnet-5-5 and carries a writer_note.
  • Videos: the graded replay.mp4 of the run viewer dataset MLL-Lab/embodied-exam-runs; runs without one (privileged runs that submitted nothing) are labelled from the log only.
  • Layout: phaseK/<suite>/<task>/trajectories/<runId>.json; the DAG stays phase2/<suite>/<task>/task_dag.json.
  • Status: 857 of 857 trajectories uploaded (427 tasks) (complete).

Phase R: unlabelled counted attempts of the seven original agents (2026-11-08)

  • Runs: the counted attempt of each of the seven original agents on each core test of the eight-agent leaderboard (456 Standard, 391 Privileged, 54 Realtime) whose task has a phase-2 DAG, when that attempt had no label under phase2/: reruns after 2026-11-06 that replaced a labelled attempt (among them the RoboDojo reruns), and attempts newer than the phase-2 run catalog. Renamed realtime runs (-standard-realtime -> -realtime, same trial) count as labelled and are not repeated.
  • Batch R10 (10 trajectories): ten core cells (six of Qwen3.8 Max, four of GPT-6 Astra, all Standard) have no counted attempt in the leaderboard; their earlier attempt on the same test is labelled here and used for the cell in the stage-level analysis only (the success counts are the leaderboard's). One of them (robodojo-make-kong-i00-standard, GPT-6 Astra) succeeded and is left out of the failure statistics.
  • Writer: Claude Sonnet 5.5 (claude-sonnet-5-5, Claude Code), as for phase K, by the owner's decision of 2026-11-08; the reviewer is Claude Sonnet 5.5 in a separate session. Prompts v3.2, the task's phase-2 DAG as published under phase2/ (not rewritten), the same task inputs, oracle rule, review rounds and validation. Each file's provenance says engine: claude, model: claude-sonnet-5-5 and carries a writer_note.
  • Layout: phaseR/<suite>/<task>/trajectories/<runId>.json; the DAG stays phase2/<suite>/<task>/task_dag.json.
  • Status: 630 of 630 trajectories uploaded (269 tasks) (complete).

Phase F: Claude Fable 5.2 (2026-11-12)

  • Runs: the counted attempt of Claude Code with Claude Fable 5.2 (run claude_code-fable_5_2-high, default seed) on each core test (privileged, standard and realtime), successes and failures alike, the same rule as for the other agents. Tests whose task has no phase-2 DAG are not labelled (no DAG is written here): embodiedswe, roboengineer and two newer Embodied Olympics - Match tasks.
  • Writer: GPT-6.1 Sol (gpt-6.1-sol, high effort) in Codex; reviewer: Claude Sonnet 5.5 in Claude Code. The same writer and reviewer as phase 2; prompts v3.2, the task's phase-2 DAG as published under phase2/ (not rewritten; dag_sha256 is that file), the same task inputs and oracle rule, review rounds and validation. Each file's provenance carries a writer_note and writer_access (ChatGPT subscription or Azure OpenAI API, same model).
  • Videos: the graded replay.mp4 of the run viewer dataset MLL-Lab/embodied-exam-runs; runs without one (privileged runs that submitted nothing) are labelled from the log only.
  • Layout: phaseF/<suite>/<task>/trajectories/<runId>.json; the DAG stays phase2/<suite>/<task>/task_dag.json.
  • Status: 855 of 855 trajectories uploaded (427 tasks) (complete).

Phase Sol: GPT-6.1 Sol (2026-11-15)

  • Runs: the counted attempt of Codex with GPT-6.1 Sol (run codex-gpt6_1_sol-high, high reasoning effort) on each core test (456 Standard, 391 Privileged, 54 Realtime), successes and failures alike, the same rule as for the other agents. Tests whose task has no phase-2 DAG are not labelled (no DAG is written here): embodiedswe, roboengineer and two newer Embodied Olympics - Match tasks. Five Realtime tests still queued and one Privileged test whose attempt ended in an infrastructure failure have no attempt to label.
  • Writer: GPT-6.1 Sol (gpt-6.1-sol, high effort) in Codex; reviewer: Claude Sonnet 5.5 in Claude Code. The same writer and reviewer as phases 2 and F; prompts v3.2, the task's phase-2 DAG as published under phase2/ (not rewritten; dag_sha256 is that file), the same task inputs and oracle rule, review rounds and validation. Each file's provenance carries a writer_note and writer_access (ChatGPT subscription). Here the writer is the model of the labelled agent; the reviewer is a different model.
  • Videos: this run is not in the run viewer dataset MLL-Lab/embodied-exam-runs. Its run records were built with the run viewer's builder (same format) from the trials, and the video is the trial's graded replay.mp4, the same file the run viewer uses. Runs without one are labelled from the log only (6 runs: 4 RoboWits and 1 DexHand Privileged runs whose replay wrote no video, 1 RoboDojo Standard run that hit the agent time limit and has only a live recording).
  • Layout: phaseSol/<suite>/<task>/trajectories/<runId>.json; the DAG stays phase2/<suite>/<task>/task_dag.json.
  • Status: 852 of 852 trajectories uploaded (427 tasks) (complete).
Downloads last month
556