Datasets:
case_id stringlengths 11 24 | edit_type stringclasses 5
values | src_video stringlengths 26 26 | prompt stringlengths 40 319 |
|---|---|---|---|
ip_object_replace_001 | ip_object_replace | source_videos/clip_001.mp4 | Replace the vase with white tulips on the counter with a LEGO Botanical Collection Orchid. |
ip_object_replace_002 | ip_object_replace | source_videos/clip_001.mp4 | Replace the black coffee machine in the background with a red KitchenAid stand mixer. |
ip_object_replace_003 | ip_object_replace | source_videos/clip_002.mp4 | Replace the potted plant on the table with a Lego Bonsai Tree while preserving the shallow depth of field. |
ip_object_replace_004 | ip_object_replace | source_videos/clip_003.mp4 | Replace the plain white boxes on the luggage cart with Rimowa aluminum suitcases. |
ip_object_replace_005 | ip_object_replace | source_videos/clip_004.mp4 | Replace the burger on the plate with a McDonald's Big Mac while preserving the lighting and camera motion. |
ip_object_replace_006 | ip_object_replace | source_videos/clip_004.mp4 | Replace the plain glass of soda with a branded Coca-Cola glass while keeping the liquid appearance consistent. |
ip_object_replace_007 | ip_object_replace | source_videos/clip_005.mp4 | Replace the small dessert on the plate with a LadurΓ©e macaron tower, keeping the plate and table setting consistent. |
ip_object_replace_008 | ip_object_replace | source_videos/clip_006.mp4 | Replace the striped shirt held by the woman with a Burberry check pattern shirt, maintaining the way it hangs on the hanger. |
ip_object_replace_009 | ip_object_replace | source_videos/clip_007.mp4 | Replace the plain white pillow on the sofa with a Louis Vuitton monogram pillow while preserving the lighting and camera motion. |
ip_object_replace_010 | ip_object_replace | source_videos/clip_008.mp4 | Replace the large pink artwork on the right wall with an Inception movie poster. |
ip_object_replace_011 | ip_object_replace | source_videos/clip_009.mp4 | Replace the grey candles on the coffee table with a Yankee Candle large jar. |
ip_object_replace_012 | ip_object_replace | source_videos/clip_010.mp4 | Replace the 'eko+' logo on the wall with a Google logo, matching the perspective and lighting. |
ip_object_replace_013 | ip_object_replace | source_videos/clip_011.mp4 | Replace the wooden rocking horse on the right with a red and yellow Little Tikes Cozy Coupe ride-on toy. |
ip_object_replace_014 | ip_object_replace | source_videos/clip_012.mp4 | Replace the green wooden dinosaur toy with a LEGO Duplo T-Rex toy, ensuring it fits naturally with the child's hand movements. |
ip_object_replace_015 | ip_object_replace | source_videos/clip_013.mp4 | Replace the animal alphabet poster on the wall with a Disney Frozen movie poster. |
ip_object_replace_016 | ip_object_replace | source_videos/clip_014.mp4 | Replace the wooden dinosaur toy in the background with a Peppa Pig plush toy. |
ip_object_replace_017 | ip_object_replace | source_videos/clip_015.mp4 | Replace the studded boots with Nike Air Jordan 1 sneakers, matching the lighting and movement of the foot. |
ip_object_replace_018 | ip_object_replace | source_videos/clip_016.mp4 | Replace the red bandana around her neck with a Burberry check silk scarf. |
ip_object_replace_019 | ip_object_replace | source_videos/clip_017.mp4 | Replace the cocktail glass on the table with a Coca-Cola classic glass bottle, maintaining the lighting and reflections on the marble surface. |
ip_object_replace_020 | ip_object_replace | source_videos/clip_018.mp4 | Replace the black backpack on the bench with a yellow Fjallraven Kanken backpack. |
ip_object_replace_021 | ip_object_replace | source_videos/clip_019.mp4 | Replace the blank canvas on the easel with Van Gogh's 'Starry Night' while maintaining the perspective and lighting of the room. |
ip_object_replace_022 | ip_object_replace | source_videos/clip_020.mp4 | Replace the acoustic guitar on the sofa with a Fender Stratocaster electric guitar. |
ip_object_replace_023 | ip_object_replace | source_videos/clip_020.mp4 | Replace the framed pictures on the wall with a vintage Beatles concert poster. |
ip_object_replace_024 | ip_object_replace | source_videos/clip_021.mp4 | Replace the yellow electric guitar with a Gibson Les Paul Standard in cherry sunburst, keeping the strap and hand positions consistent. |
ip_object_replace_025 | ip_object_replace | source_videos/clip_022.mp4 | Replace the glass bottle lying on the blanket with a Veuve Clicquot champagne bottle. |
ip_object_replace_026 | ip_object_replace | source_videos/clip_023.mp4 | Replace the black briefcase on the ground next to the man with a Tumi Alpha 3 Briefcase. |
ip_object_replace_027 | ip_object_replace | source_videos/clip_024.mp4 | Replace the plain grey back cushion on the chair with a Marimekko Unikko patterned cushion. |
ip_object_replace_028 | ip_object_replace | source_videos/clip_025.mp4 | Replace the champagne flutes with Stella Artois chalice glasses. |
ip_object_replace_029 | ip_object_replace | source_videos/clip_026.mp4 | Replace the wooden directional signpost with a vintage Route 66 sign post, ensuring the shadows on the sand match the new object. |
ip_object_replace_030 | ip_object_replace | source_videos/clip_027.mp4 | Replace the overwater bungalows in the distance with a Disney Cruise Line ship anchored in the water. |
ip_object_replace_031 | ip_object_replace | source_videos/clip_028.mp4 | Replace the blank canvas inside the left picture frame with a vintage Coca-Cola poster. |
ip_object_replace_032 | ip_object_replace | source_videos/clip_029.mp4 | Replace the top-most photograph of the woman on the beach with a print of 'The Great Wave off Kanagawa' while maintaining the same size and tape placement. |
ip_object_replace_033 | ip_object_replace | source_videos/clip_030.mp4 | Replace the black fencing mask held under her arm with a Darth Vader helmet. |
ip_object_replace_034 | ip_object_replace | source_videos/clip_031.mp4 | Replace the red patterned blanket with a Pendleton Yakima Camp Blanket, maintaining the folds and the woman sitting on it. |
ip_object_replace_035 | ip_object_replace | source_videos/clip_032.mp4 | Replace the glass water bottle in the foreground with a Hydro Flask water bottle, maintaining the lighting and reflections on the table. |
ip_object_replace_036 | ip_object_replace | source_videos/clip_033.mp4 | Replace the plain building blocks with a stack of colorful LEGO Duplo bricks, ensuring the child's hands interact naturally with the new bricks. |
ip_object_replace_037 | ip_object_replace | source_videos/clip_034.mp4 | Replace the round mirror on the wall with a framed Star Wars movie poster. |
ip_object_replace_038 | ip_object_replace | source_videos/clip_035.mp4 | Replace the glass of water on the desk with a Starbucks iced coffee cup. |
ip_object_replace_039 | ip_object_replace | source_videos/clip_036.mp4 | Replace the glass bowl of salad on the left side of the counter with a red Le Creuset Dutch oven. |
ip_object_replace_040 | ip_object_replace | source_videos/clip_037.mp4 | Replace the plain red sweater with a red Adidas sweatshirt featuring a prominent logo. |
ip_object_replace_041 | ip_object_replace | source_videos/clip_038.mp4 | Replace the silver metal mug the woman is holding with a Yeti Rambler Mug. |
ip_object_replace_042 | ip_object_replace | source_videos/clip_039.mp4 | Replace the book the boys are reading with 'The Very Hungry Caterpillar' book. |
ip_object_replace_043 | ip_object_replace | source_videos/clip_040.mp4 | Replace the small potted plant on the left with a Lego flower bouquet. |
ip_object_replace_044 | ip_object_replace | source_videos/clip_041.mp4 | Replace the glowing sphere on the table with an Amazon Echo Dot 4th Gen. |
ip_object_replace_045 | ip_object_replace | source_videos/clip_042.mp4 | Replace the glass mug on the table with a Nespresso glass coffee cup. |
ip_object_replace_046 | ip_object_replace | source_videos/clip_043.mp4 | Replace the laptop on the table with a silver Apple MacBook Pro. |
ip_object_replace_047 | ip_object_replace | source_videos/clip_044.mp4 | Replace the clear plastic cups the people are holding with Ben & Jerry's ice cream mini cups. |
ip_object_replace_048 | ip_object_replace | source_videos/clip_045.mp4 | Replace the white teddy bear with a Paddington Bear plush toy, ensuring it fits naturally in the man's hands and follows the motion. |
ip_object_replace_049 | ip_object_replace | source_videos/clip_046.mp4 | Replace the glass of amber liquid with a bottle of Ito En green tea. |
ip_object_replace_050 | ip_object_replace | source_videos/clip_047.mp4 | Replace the light blue feather held by the woman with a Gucci hand fan. |
ip_object_replace_051 | ip_object_replace | source_videos/clip_048.mp4 | Replace the large smooth stones on the right side of the person with a silver Rimowa Classic Cabin suitcase. |
ip_object_replace_052 | ip_object_replace | source_videos/clip_049.mp4 | Replace the plain laptop on the desk with a silver Apple MacBook Pro. |
ip_object_replace_053 | ip_object_replace | source_videos/clip_050.mp4 | Replace the white toaster on the left counter with a Smeg retro 2-slice toaster. |
ip_object_replace_054 | ip_object_replace | source_videos/clip_051.mp4 | Replace the small white cup on the desk with a Nespresso coffee mug. |
ip_object_replace_055 | ip_object_replace | source_videos/clip_052.mp4 | Replace the black fire pit in the center with a stainless steel Solo Stove Bonfire. |
ip_object_replace_056 | ip_object_replace | source_videos/clip_053.mp4 | Replace the white armchair in the background with a classic Eames Lounge Chair. |
ip_object_add_001 | ip_object_add | source_videos/clip_054.mp4 | Add a framed Starry Night by Vincent van Gogh poster to the empty white paneled wall behind the person. |
ip_object_add_002 | ip_object_add | source_videos/clip_055.mp4 | Add a Fisher-Price Rock-a-Stack toy on the patterned rug next to the woman and child. |
ip_object_add_003 | ip_object_add | source_videos/clip_056.mp4 | Add a framed Star Wars movie poster to the empty space on the green wall behind the parents. |
ip_object_add_004 | ip_object_add | source_videos/clip_057.mp4 | Add a LEGO Darth Vader minifigure on the table next to the wine glass. |
ip_object_add_005 | ip_object_add | source_videos/clip_058.mp4 | Add a bottle of Hendrick's Gin on the white counter to the left of the tray as it is being placed. |
ip_object_add_006 | ip_object_add | source_videos/clip_059.mp4 | Add a Whole Foods Market logo to the wall behind the woman. |
ip_object_add_007 | ip_object_add | source_videos/clip_060.mp4 | Add a Smeg retro toaster on the empty wooden countertop to the right of the woman. |
ip_object_add_008 | ip_object_add | source_videos/clip_061.mp4 | Add a red KitchenAid stand mixer to the empty counter space on the left side of the frame. |
ip_object_add_009 | ip_object_add | source_videos/clip_062.mp4 | Add a Jellycat Bashful Bunny plush toy sitting on the right side of the grey sofa. |
ip_object_add_010 | ip_object_add | source_videos/clip_008.mp4 | Add a Tom Ford coffee table book on the wooden coffee table, ensuring it matches the lighting and shadows of the scene. |
ip_object_add_011 | ip_object_add | source_videos/clip_063.mp4 | Add a Sensodyne toothpaste box on the reception desk next to the small plant. |
ip_object_add_012 | ip_object_add | source_videos/clip_064.mp4 | Add a standard Rubik's Cube on the empty space on the table to the left of the child. |
ip_object_add_013 | ip_object_add | source_videos/clip_065.mp4 | Add a Tonka yellow dump truck toy on the floor to the right of the building blocks. |
ip_object_add_014 | ip_object_add | source_videos/clip_066.mp4 | Add a Jellycat Bashful Bunny plush toy sitting inside the woven basket in the background. |
ip_object_add_015 | ip_object_add | source_videos/clip_067.mp4 | Add an Iron Man Funko Pop figure on the desk next to the left speaker, ensuring it matches the scene's lighting. |
ip_object_add_016 | ip_object_add | source_videos/clip_068.mp4 | Add a Batman Funko Pop figure on the desk to the left of the keyboard, ensuring it matches the scene's purple and orange lighting. |
ip_object_add_017 | ip_object_add | source_videos/clip_017.mp4 | Add a pair of Ray-Ban Wayfarer sunglasses resting on the marble table next to the cocktail glass. |
ip_object_add_018 | ip_object_add | source_videos/clip_069.mp4 | Add a RE/MAX hot air balloon flying in the sky above the mountains. |
ip_object_add_019 | ip_object_add | source_videos/clip_020.mp4 | Add a Marshall Emberton bluetooth speaker on the wooden coffee table next to the papers. |
ip_object_add_020 | ip_object_add | source_videos/clip_021.mp4 | Add a glowing Fender neon sign on the brick wall behind the singer. |
ip_object_add_021 | ip_object_add | source_videos/clip_070.mp4 | Add a bag of Blue Bottle Coffee beans on the table next to the pour-over coffee maker. |
ip_object_add_022 | ip_object_add | source_videos/clip_071.mp4 | Add a black Casio G-Shock GA-2100-1A1ER watch to the person's wrist as they reach for the flyer. |
ip_object_add_023 | ip_object_add | source_videos/clip_072.mp4 | Add a Kikkoman soy sauce bottle on the white countertop next to the wooden cutting board. |
ip_object_add_024 | ip_object_add | source_videos/clip_033.mp4 | Add a classic Steiff teddy bear sitting on the pillow to the left of the child. |
ip_object_add_025 | ip_object_add | source_videos/clip_073.mp4 | Add a red KitchenAid stand mixer on the counter in the background. |
ip_object_add_026 | ip_object_add | source_videos/clip_074.mp4 | Add a Starbucks iced coffee cup on the desk to the left of the laptop. |
ip_object_add_027 | ip_object_add | source_videos/clip_075.mp4 | Add a luxury hotel logo to the chest area of the white bathrobe. |
ip_object_add_028 | ip_object_add | source_videos/clip_076.mp4 | Add a Fisher-Price Rock-a-Stack toy on the empty space on the sofa to the left of the child. |
ip_object_add_029 | ip_object_add | source_videos/clip_077.mp4 | Add a glowing Heineken neon sign to the empty wall space behind the man on the right. |
ip_object_add_030 | ip_object_add | source_videos/clip_078.mp4 | Add an ASPCA logo poster to the empty grey wall behind the volunteers. |
ip_object_add_031 | ip_object_add | source_videos/clip_079.mp4 | Add a classic Coca-Cola logo to the red background behind the woman. |
ip_object_add_032 | ip_object_add | source_videos/clip_080.mp4 | Add a Squishmallows plush toy on the sofa next to the woman. |
ip_object_add_033 | ip_object_add | source_videos/clip_081.mp4 | Add a bag of King Arthur Baking Company flour on the counter to the left of the girl. |
ip_object_add_034 | ip_object_add | source_videos/clip_052.mp4 | Add a white Yeti Tundra cooler on the ground next to the wooden chair on the left. |
ip_object_add_035 | ip_object_add | source_videos/clip_082.mp4 | Add a closed silver MacBook Pro to the empty space on the wooden table between the woman on the left and the child. |
ip_object_add_036 | ip_object_add | source_videos/clip_083.mp4 | Add a vintage Louis Vuitton trunk resting on the sand in the foreground, fitting the high-fashion editorial style of the scene. |
ip_object_add_037 | ip_object_add | source_videos/clip_084.mp4 | Add a Fender neon sign on the plain wall behind the children. |
ip_object_add_038 | ip_object_add | source_videos/clip_085.mp4 | Add a RE/MAX hot air balloon floating in the clear sky above the mountains. |
ip_object_add_039 | ip_object_add | source_videos/clip_086.mp4 | Add a Land Rover Defender parked on the sand dunes in the background. |
ip_object_add_040 | ip_object_add | source_videos/clip_087.mp4 | Add a Goodyear blimp flying in the empty blue sky to the left of the kite. |
ip_background_change_001 | ip_background_change | source_videos/clip_088.mp4 | Change the background view through the window to a realistic New York City skyline at sunset while keeping the interior and people unchanged. |
ip_background_change_002 | ip_background_change | source_videos/clip_089.mp4 | Change the background to Andy's room from Toy Story, featuring the iconic cloud wallpaper, while keeping the girl and her toy unchanged. |
ip_background_change_003 | ip_background_change | source_videos/clip_090.mp4 | Change the background to Central Park's Sheep Meadow on a sunny day, keeping the couple and their picnic setup unchanged. |
ip_background_change_004 | ip_background_change | source_videos/clip_025.mp4 | Change the background to a view of the Eiffel Tower in Paris, maintaining the depth of field. |
AgentEdit-Bench is the video-editing benchmark introduced with Aurora: Unified Video Editing with a Tool-Using Agent (NeurIPS 2026). Its 150 cases pair a source clip with a request that leaves out something a video editor needs: a reference image for a named entity, a mask for the object to remove, or a direct statement of the edit. Edited videos are scored by a VLM judge.
Aurora
Aurora is an agentic video-editing framework. Given a source video and a natural-language request, a tool-using vision-language agent first resolves ambiguity: it rewrites the request into an editor-facing instruction, decides the edit type, optionally retrieves a reference image from the web, and optionally grounds a target object with a mask. A unified video diffusion editor then performs the actual edit.
The editor uses a WAN2.2-TI2V-5B diffusion transformer, a WAN2.2 VAE, and a frozen Qwen3.5-4B multimodal LLM; the agent is a Qwen3-VL-8B-Instruct planner with an Aurora LoRA adapter. One editor checkpoint supports source-conditioned editing (s2v), video-to-video editing (v2v), and reference-conditioned editing (sv2v).
News
- 2026-09: Aurora is accepted to NeurIPS 2026.
- 2026-06: Code, model weights, and training data are released.
- 2026-05: The paper is released on arXiv.
The benchmark
Real editing requests rarely arrive in a model-ready form. Each case is built around a request that is underspecified along one of two axes:
- Visual underspecification. The request names a specific real-world entity (a brand, product, character, landmark, or artwork) without a reference image, or names an object to remove without a mask.
- Textual underspecification. The edit is described only indirectly, so the concrete editing goal has to be inferred.
Edit type (edit_type) |
Cases | What the request leaves out | Example request |
|---|---|---|---|
IP replacement (ip_object_replace) |
56 | Appearance of the named entity | Replace the white armchair in the background with a classic Eames Lounge Chair. |
IP addition (ip_object_add) |
40 | Appearance of the named entity | Add a Goodyear blimp flying in the empty blue sky to the left of the kite. |
IP background change (ip_background_change) |
12 | Appearance of the named place | Change the background view through the window to a realistic New York City skyline at sunset while keeping the interior and people unchanged. |
Reasoning (reasoning) |
17 | A direct statement of the edit | Turn the lake into a perfect mirror for the passing clouds above. |
Removal (removal) |
25 | Location of the target and of its physical effects | Remove the white tennis racket resting vertically on the court. You must also completely erase the intricate shadow cast by its frame and strings onto the ground, replacing it with the natural, continuous texture of the clay surface. |
"IP" stands for a named real-world entity such as a brand, product, character, or artwork.
- Reasoning requests use four linguistic styles (
reasoning_style): referential (5), property (5), counterfactual (5), and elliptical (2). - In 18 of the 25 removal cases the target carries physical effects that must
disappear with it (
physical_effects, multi-label): shadow cast (14), mirror reflection (4), water reflection (4), complex occlusion (4), water ripple (3), light cast (2), and footprints (1).
The same clips support three evaluation protocols:
- Full framework. The Aurora agent plans each edit and the Aurora editor renders it.
- Agent transfer. The Aurora agent's plan drives another video editing model.
- Instruction only. Any video editing model edits from the raw request alone.
Dataset structure
AgentEdit-Bench/
βββ prompts.jsonl # 150 cases: what a system under test receives
βββ cases.json # 150 cases: the same inputs plus judge annotations
βββ source_videos/ # 140 source clips (MP4)
The clips are 102 landscape and 38 portrait videos, from 480p to 4K, and 1.8 to 30.5 s long (median 5.9 s). Nine clips are shared by two or three cases with different requests. All paths are relative to the dataset root.
prompts.jsonl holds one case per line:
{"case_id": "ip_object_replace_001", "edit_type": "ip_object_replace", "src_video": "source_videos/clip_001.mp4", "prompt": "Replace the vase with white tulips on the counter with a LEGO Botanical Collection Orchid."}
cases.json is a list with the same fields plus the annotations the judge
uses:
{
"case_id": "ip_object_replace_001",
"edit_type": "ip_object_replace",
"src_video": "source_videos/clip_001.mp4",
"prompt": "Replace the vase with white tulips on the counter with a LEGO Botanical Collection Orchid.",
"target_entity": "LEGO Botanical Collection Orchid",
"edit_region": "the vase with white tulips on the left side of the counter",
"web_query": "LEGO Botanical Collection Orchid product photo"
}
| Field | Description |
|---|---|
case_id |
Unique id, <edit_type>_<NNN>. |
edit_type |
ip_object_replace, ip_object_add, ip_background_change, reasoning, or removal. |
src_video |
Source clip. |
prompt |
The user request, exactly as the system under test receives it. |
target_entity |
The named entity an IP edit must show. Empty for reasoning and removal. |
edit_region |
The region the edit should stay within. Empty when not annotated. |
web_query |
A canonical image-search query for target_entity. Empty for reasoning and removal. |
reasoning_style |
Reasoning cases only: referential, property, counterfactual, or elliptical. |
physical_effects |
Removal cases only: effects tied to the target, such as shadow_cast or water_reflection. Empty when none are annotated. |
A system under test should receive only
promptand the source clip.target_entity,edit_region, andweb_queryare judge annotations; giving them to an editor or agent removes the underspecification the benchmark measures.
Usage
Download
hf download yeates/AgentEdit-Bench --repo-type dataset --local-dir AgentEdit-Bench
from huggingface_hub import snapshot_download
bench_dir = snapshot_download("yeates/AgentEdit-Bench", repo_type="dataset")
Run Aurora
The inference and scoring scripts are part of the Aurora code; run the commands below from its repository root. First, let the Aurora agent plan every case. Web image search uses the Serper API.
export BENCH=/abs/path/to/AgentEdit-Bench
# Convert the benchmark into the agent's case format.
python - <<'EOF'
import json, os
bench = os.environ["BENCH"]
with open(os.path.join(bench, "prompts.jsonl")) as src, open("agentedit_cases.jsonl", "w") as dst:
for line in src:
case = json.loads(line)
dst.write(json.dumps({
"bench_id": case["case_id"],
"edit_type": case["edit_type"],
"prompt": case["prompt"],
"video_path": os.path.join(bench, case["src_video"]),
}) + "\n")
EOF
export SERPER_KEY_ID=<your-serper-key>
python -m aurora.agent \
--custom_cases_jsonl agentedit_cases.jsonl \
--custom_only \
--mask_backend grounded_sam \
--out_dir "$PWD/agent_out" \
--serper_api_key "$SERPER_KEY_ID"
Then render the plans with the Aurora editor, using the settings from the paper:
python evaluation/agentedit_bench_infer.py \
--bench_dir "$BENCH" \
--agent_records_jsonl "$PWD/agent_out/agent_pipeline_records.jsonl" \
--use_mask_overlay \
--num_gpus 8 \
--max_pixels 921600 \
--num_frames 81 \
--cfg_scale 2.0 \
--image_cfg_scale 1.25 \
--out_dir "$PWD/results/aurora"
Keep --out_dir absolute for the agent: the editor resolves relative asset
paths against --bench_dir. For the instruction-only protocol, drop
--agent_records_jsonl and --use_mask_overlay. For agent transfer, give your
editor each record's final_payload (refined_text_instruction, the
search_image path, and the object_mask path, either of which may be
false) in place of the raw request.
Score
The scorer reads one edited video per case from
<results_dir>/<case_id>/generate.mp4. To score another editor, save its
outputs the same way. If the editor resizes or trims its input, also save the
matching input as <results_dir>/<case_id>/source.mp4 so that the BEFORE and
AFTER frames line up.
export GEMINI_API_KEY=<your-gemini-key>
python evaluation/agentedit_bench_score.py \
--bench_dir "$BENCH" \
--results_dir "$PWD/results/aurora" \
--api openai \
--base_url https://generativelanguage.googleapis.com/v1beta/openai/ \
--api_key "$GEMINI_API_KEY" \
--model gemini-2.5-pro
This runs the judge as in the paper: Gemini 2.5 Pro at temperature 0, called
through its OpenAI-compatible endpoint, without web search. Scores are written
to <results_dir>/scores.json. The values
summary.by_edit_type.<edit_type>.avg_pct_on_1 and summary.overall_pct_on_1,
multiplied by 100, are the numbers reported in the table below.
Evaluation protocol
Gemini 2.5 Pro at temperature 0, without web search, compares three BEFORE/AFTER frame pairs per case, taken at frames 0, k, and 2k of the edited clip with k = βN/3β, and scores each axis from 0 to 3.
| # | Axis | What it measures | Applies to |
|---|---|---|---|
| 1 | Instruction Following | Whether the requested edit happened in the named region with the named target | All types |
| 2 | Edit Region Localization | Whether the change stays inside the specified region | All types |
| 3 | Source Preservation | Whether unedited regions, motion, geometry, and lighting are preserved | All types |
| 4 | Visual Quality | Realism, lighting and shadow match, scale and perspective, absence of paste-on artifacts | All types |
| 5 | Temporal Consistency | Whether the edited content would stay stable across the clip | All types |
| 6 | IP Presence | Whether the named entity is visible and recognizable in roughly the right region | IP edits |
| 7 | IP Identity Match | Whether the visible entity matches the specific real-world identity (brand colors, logo, signature shape) | IP edits |
IP edits are scored on all seven axes (maximum 21); reasoning and removal cases on axes 1 to 5 (maximum 15). A removal output that replaces the target with a new object scores at most 1 on Instruction Following. Axis scores are averaged over the three frames, each case is normalized by its maximum, and per-type and overall scores are means over cases, reported as percentages.
Results
Table 2 of the paper. β means the editor is paired with the Aurora VLM agent; β means it edits from the raw request.
| Method | Aurora agent | IP Replace | IP Add | IP Background | Reasoning | Removal | Overall |
|---|---|---|---|---|---|---|---|
| UniVideo | β | 67.1 | 63.4 | 65.6 | 77.3 | 66.5 | 67.0 |
| UniVideo | β | 80.3 | 81.3 | 66.8 | 74.6 | 68.1 | 76.8 |
| Kiwi-Edit | β | 66.7 | 65.1 | 86.6 | 79.5 | 69.0 | 69.7 |
| Kiwi-Edit | β | 71.6 | 73.1 | 78.3 | 84.3 | 57.8 | 71.7 |
| Aurora | β | 73.0 | 68.0 | 71.2 | 86.4 | 83.0 | 74.7 |
| Aurora | β | 89.6 | 85.3 | 90.1 | 88.1 | 86.8 | 87.9 |
Source attribution and licenses
AgentEdit-Bench is intended for research evaluation. The source clips are third-party footage, and each keeps its original license; consult the source before redistribution or commercial use.
- Pexels. The 123 clips used by the IP and removal cases come from the Pexels stock library and are covered by the Pexels License.
- Public video-editing datasets. The 17 reasoning clips are source videos from Ditto-1M (9), OpenVE-3M (6), ReCo-Data (1), and EffectErase (1).
The editing requests and all annotations in prompts.jsonl and cases.json
were created and curated by the Aurora authors.
Citation
@inproceedings{yu2026aurora,
title={Aurora: Unified Video Editing with a Tool-Using Agent},
author={Yu, Yongsheng and Zeng, Ziyun and Xiao, Zhiyuan and Zhou, Zhenghong and Hua, Hang and Xiong, Wei and Luo, Jiebo},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2026}
}
- Downloads last month
- 62