Datasets:

ArXiv:
Libraries:
Datasets
License:
Dataset Viewer
Auto-converted to Parquet Duplicate
video
video
1.4
98.1
dataset
stringclasses
6 values
split
stringclasses
1 value
trajectory_name
stringlengths
5
94
subtask_index
int64
0
0
language_instruction
stringlengths
9
685
initial_object_box
listlengths
4
4
target_object_box
listlengths
4
4
img_width
int64
214
1.28k
img_height
int64
172
720
native_fps
float64
5
30
effective_fps
float64
5
30
num_frames
int64
13
2.94k
num_frames_original
int64
13
2.94k
frame_indices
listlengths
13
2.94k
is_bimanual
bool
2 classes
annotator
stringclasses
1 value
proprio
stringlengths
1.63k
1.26M
proprio_keys
stringclasses
4 values
proprio_dims
stringclasses
6 values
galaxea
validation
Replace_The_Toilet_Paper_Roll_20250805_012/0/5/1
0
pick up the empty paper roll with the right hand and place it in the trash bin.
[ 652, 362, 704, 428 ]
[ 702, 505, 842, 682 ]
1,280
720
30
30
216
216
[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54...
true
human_gt
{"left_gripper_state": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0...
left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_pos
{"left_gripper_state": [1], "left_eef_state": [7], "left_joint_pos": [6], "right_gripper_state": [1], "right_eef_state": [7], "right_joint_pos": [6]}
galaxea
validation
Arrange_The_Tray_20250725_010/0/35/4
0
pick up the spoon with the right hand. place the spoon back with the right hand.
[ 752, 430, 840, 620 ]
[ 913, 406, 1040, 554 ]
1,280
720
30
30
100
100
[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54...
true
human_gt
{"left_gripper_state": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0...
left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_pos
{"left_gripper_state": [1], "left_eef_state": [7], "left_joint_pos": [6], "right_gripper_state": [1], "right_eef_state": [7], "right_joint_pos": [6]}
galaxea
validation
Arrange_The_Fruits_20250822_013/0/10/2
0
pick up the starfruit with your right hand and place it on the plate.
[ 906, 580, 1265, 719 ]
[ 506, 414, 717, 622 ]
1,280
720
30
30
198
198
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,3(...TRUNCATED)
true
human_gt
"{\"left_gripper_state\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0(...TRUNCATED)
"left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_po(...TRUNCATED)
"{\"left_gripper_state\": [1], \"left_eef_state\": [7], \"left_joint_pos\": [6], \"right_gripper_sta(...TRUNCATED)
galaxea
validation
Desktop_Garbage_Decluttering_20250712_006/0/87/0
0
pick up the plastic bottle on the black table with your right hand.
[ 751, 32, 953, 315 ]
[ 865, 3, 1106, 255 ]
1,280
720
30
30
74
74
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,3(...TRUNCATED)
true
human_gt
"{\"left_gripper_state\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0(...TRUNCATED)
"left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_po(...TRUNCATED)
"{\"left_gripper_state\": [1], \"left_eef_state\": [7], \"left_joint_pos\": [6], \"right_gripper_sta(...TRUNCATED)
galaxea
validation
Toast_Bread_20250618_001/0/66/1
0
press the toaster switch with your left hand.
[ 270, 488, 352, 533 ]
[ 253, 294, 538, 591 ]
1,280
720
30
30
154
154
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,3(...TRUNCATED)
true
human_gt
"{\"left_gripper_state\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0(...TRUNCATED)
"left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_po(...TRUNCATED)
"{\"left_gripper_state\": [1], \"left_eef_state\": [7], \"left_joint_pos\": [6], \"right_gripper_sta(...TRUNCATED)
galaxea
validation
Open_And_Close_The_Notebook20250628_002/0/89/4
0
hook the middle of the computer screen upward with your left hand.
[ 499, 466, 838, 668 ]
[ 481, 439, 827, 645 ]
1,280
720
30
30
67
67
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,3(...TRUNCATED)
true
human_gt
"{\"left_gripper_state\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0(...TRUNCATED)
"left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_po(...TRUNCATED)
"{\"left_gripper_state\": [1], \"left_eef_state\": [7], \"left_joint_pos\": [6], \"right_gripper_sta(...TRUNCATED)
galaxea
validation
Cooking_Food_In_An_Air_Fryer_20250801_012/0/23/5
0
hold the fryer handle with your right hand and remove the fryer.
[ 639, 234, 821, 495 ]
[ 661, 427, 880, 701 ]
1,280
720
30
30
180
180
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,3(...TRUNCATED)
true
human_gt
"{\"left_gripper_state\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0(...TRUNCATED)
"left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_po(...TRUNCATED)
"{\"left_gripper_state\": [1], \"left_eef_state\": [7], \"left_joint_pos\": [6], \"right_gripper_sta(...TRUNCATED)
galaxea
validation
Open_And_Close_Nightstand_Door_20250625_002/0/51/6
0
fully open the left sliding door with the left hand.
[ 114, 0, 1279, 515 ]
[ 907, 155, 1279, 544 ]
1,280
720
30
30
187
187
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,3(...TRUNCATED)
true
human_gt
"{\"left_gripper_state\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0(...TRUNCATED)
"left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_po(...TRUNCATED)
"{\"left_gripper_state\": [1], \"left_eef_state\": [7], \"left_joint_pos\": [6], \"right_gripper_sta(...TRUNCATED)
galaxea
validation
Connect_Router_Cables_20250625_002/0/100/1
0
insert the charger head into the power strip with the left hand.
[ 198, 512, 323, 682 ]
[ 458, 281, 531, 345 ]
1,280
720
30
30
285
285
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,3(...TRUNCATED)
true
human_gt
"{\"left_gripper_state\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0(...TRUNCATED)
"left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_po(...TRUNCATED)
"{\"left_gripper_state\": [1], \"left_eef_state\": [7], \"left_joint_pos\": [6], \"right_gripper_sta(...TRUNCATED)
galaxea
validation
Water_Dispenser_Connected_To_Water20250707_005/0/52/1
0
fully open the left-hand water dispenser cabinet door.
[ 513, 432, 628, 719 ]
[ 402, 420, 626, 621 ]
1,280
720
30
30
191
191
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,3(...TRUNCATED)
true
human_gt
"{\"left_gripper_state\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0(...TRUNCATED)
"left_eef_state,left_gripper_state,left_joint_pos,right_eef_state,right_gripper_state,right_joint_po(...TRUNCATED)
"{\"left_gripper_state\": [1], \"left_eef_state\": [7], \"left_joint_pos\": [6], \"right_gripper_sta(...TRUNCATED)
End of preview. Expand in Data Studio

IA-bench ( Interaction-Aware Bench)

Human ground-truth annotations of the interacted object for robot manipulation subtasks. Each sample is one subtask: the full subtask video clip, the gripper proprioception aligned 1:1 to those frames, the language instruction, and two boxes: initial_object_box (object on the first frame) and target_object_box (object on the last frame). Boxes are pixel [x1, y1, x2, y2].

Configs

from datasets import load_dataset
ds = load_dataset("irl-kit/IA-bench", "bridge_lerobot", split="validation")
ds = load_dataset("irl-kit/IA-bench", "all", split="test")
dataset test validation
agibot_world 264 187
bridge_lerobot 321 134
droid_lerobot 269 138
galaxea - 62
oxe_lerobot 424 14
robocoin - 67

The source repository stores VideoFolder metadata as metadata.jsonl, which is loaded automatically by load_dataset. The all config contains 1,880 unique samples. Hugging Face's headline row total sums overlapping configs and therefore does not represent the number of unique samples.

Per-frame fields

  • video: the full subtask clip (mp4, native resolution, all frames).
  • native_fps / effective_fps: source fps and the clip fps (equal unless a frame cap is applied). frame_indices gives original frame indices (timestamp = index / native_fps); proprio is aligned 1:1 with the frames.
  • proprio (JSON, decode with json.loads): unified, raw (un-normalized) per-frame proprioception. Consistent field names across datasets: gripper_state (open/close, inverted to a common convention), eef_state (6/7-dim end-effector pose), joint_pos (arm joint positions). Bimanual datasets prefix fields with left_/right_. proprio_dims gives the per-field feature dimension; proprio_keys lists the present fields. Availability/dim varies by embodiment (e.g. OXE exposes gripper only; AgiBot eef is xyz-only).

Evaluation

eval_ia_bench.py scores predicted start (initial_object_box) and target (target_object_box) boxes against the GT and reports the paper metrics: acc@IoU, AUROC, AURC, E-AURC, cov@90, cov@95, R@90, R@95 (per dataset + overall). A prediction row needs dataset, trajectory_name, subtask_index, the two boxes, and a confidence score.

python eval_ia_bench.py --predictions preds.jsonl --gt-repo irl-kit/IA-bench

Citation

If you use IA-bench, please cite:

@misc{blank2026sparcreliablespatialannotations,
      title={SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale}, 
      author={Nils Blank and Paul Mattes and Maximilian Xiling Li and Jakub Suliga and Thomas Roth and Moritz Reuss and Pankhuri Vanjani and Rudolf Lioutikov},
      year={2026},
      eprint={2606.13497},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2606.13497}, 
}
Downloads last month
398

Collection including irl-kit/IA-Bench

Paper for irl-kit/IA-Bench