Dataset Viewer
Auto-converted to Parquet Duplicate
chrome
list
gimp
list
libreoffice_calc
list
libreoffice_impress
list
libreoffice_writer
list
multi_apps
list
os
list
thunderbird
list
vlc
list
vs_code
list
[ "0cd34de7-c200-43f6-bbdc-54e4f29173e6", "139f088a-6807-4a85-868a-d3916e4fb4ab", "17f11329-1d1b-456f-9e6e-acec9752960c", "1e987a1d-804c-4a57-a877-c89e14b96172", "27308370-ac79-4fd1-a979-75f846851e87", "2acf70e9-5ce7-4500-873f-2f850b100208", "33497f7f-2548-4ebf-92f1-a1d3abb6f63a", "3ca6c72f-7549-4618-98...
[ "003f4ab4-721c-4135-a3b8-7c05b28e8c53", "041b86a3-1a5f-41ac-bc59-6c380f41d4a4", "0686c020-5bf5-43eb-9201-051fef806bf7", "179c3e5e-f1e5-43f5-b2ae-e59d626a6b90", "23b5e153-9518-48e3-9148-fc5a1a0ab63d", "2cb07954-8220-4010-923b-25d32264f527", "366aa110-c775-4879-af1b-5b1ee595cfa2", "38623924-94d4-4029-bd...
[ "03b56137-8823-4e2a-95b1-753ec589d7b9", "05b0c883-625d-4fec-9d74-2012c93ca9b0", "12949b33-c561-4d2a-953e-e7961755c333", "1a4235a4-a06a-4264-9eb9-e3a0c2b1f1ff", "21b544f3-7541-4fa8-b397-7deb2d3e836e", "21f8076e-b62b-4aeb-9bbc-a99acb4b5b11", "22da0544-70d2-44fe-bd9d-282520bb5f0a", "2bad1da1-e981-47a6-98...
[ "04de74cf-e3b1-43d5-b219-fd6f0bbffd1b", "04e1cb16-2127-4f2d-a6c5-8cc08aaa99e6", "0a0eb6f2-3b2b-48be-b135-b68c62539efe", "0a42ca7d-e173-4302-add1-598b8c0ac7d4", "123c6cd1-42d0-48ae-8a7e-14dad55c2545", "1fb9805b-3b8f-4d5f-8480-79641da3a81b", "2a06f4e5-62e5-4a5c-9631-f510220d6e01", "2dab3dde-96de-4f7d-ab...
[ "00e29c5a-1fbb-4aae-86b4-84186e7b6714", "0d4d369c-3bdd-4497-9218-2204c0d8e0c2", "0ea5e24e-35e8-46b3-8c2f-c77b365983b7", "1f398f9b-2bbd-4b81-ab36-8c7becdd8390", "2657a92b-35f2-49f7-bca8-2b1e185b03f2", "269bf124-ad55-4714-90dc-04889b9a8832", "27993ee3-47ee-47ee-b434-e1ed97fbe4e9", "377c7d07-a0d2-4e74-93...
[ "05c5e0b4-6b1a-4a72-a64d-e954a51a42c7", "1148ac53-10af-468d-977d-54e04aadd697", "11ae0817-a112-487b-9d9d-4415cccedc36", "13dcab6a-1220-4a0e-9259-aa4fac199123", "1ac9f041-ca0a-4d9d-97de-55313edae417", "1feb94fc-5a29-4005-94fe-58116d7dd8c0", "21a0f463-b842-4585-a277-7a8e7711d893", "3064b551-9283-41a0-9d...
[ "04f701f9-2c41-4886-98b2-edeb135c83b2", "08e9f469-dc86-4936-8c9f-fb087d4adf80", "0c76f872-f527-48c8-b99c-a2d5bed0650d", "0dcc75dd-7a9a-4421-a2e0-f404337c1b42", "0df8972f-d1ee-4973-8d64-2a008f519eb5", "1730da0e-53e5-4b91-8fae-59ffd566de37", "1d766da2-da2c-47cd-a81c-115011616c49", "1ff3bf87-79ea-4c2f-8b...
[ "1465f0c5-c304-4dfc-adfc-0a123b0e0982", "180ba527-7988-43c0-993c-6b1f300c390e", "2254590f-1ee4-461c-a077-4c979bae9c66", "3a8feb24-527f-4b26-a313-3e77e4e7d0ab", "4fb3c4a6-0f7d-4d79-9e72-aeda323af43f", "5b3f71e0-fbc4-475a-a7fa-94c9bbf7094a", "760bb839-2f54-4921-92a1-574edf4543ba", "7f217aea-211c-4c33-a5...
[ "0f77e8ca-e796-44e6-b56e-f0ffbb4e1b36", "1752e7c8-4402-4865-b59a-c1d690c28f28", "23315cbb-d5ce-4931-96c4-90e1d4aedfbc", "2ba4029c-be74-4bdb-8acf-0a090fe67c18", "33ef23db-84bc-461d-a5bd-11e18f04788b", "4140c1a7-1e82-4bea-9622-4a9171420df7", "49f54f4d-5729-4d32-af8c-5bc7abc829c5", "68c73326-46a9-490b-b3...
[ "033e9f78-776d-4a45-9dfd-bd433547b79b", "07d26a90-ea78-48b9-ad0d-9f6b38e2393c", "08446086-f702-45df-8f19-13fcd11ff8c7", "0c19e825-0b15-4147-8ff4-9e5c34f4586c", "1fce5281-3c90-44b4-b8c1-f8e812e01601", "23d08e4c-e5dc-460e-9a85-a01cb6acf79d", "2938f1a6-b54f-4386-81d1-ea347cebfb23", "2b071642-1394-4cd6-b4...

NightWatch

NightWatch training data

Training data of NightWatch-0.8B, the watcher of NightWatch (Don't Sleep, Keep Watch: Proactive Perception for Computer-Use Agents). Each example is one decision point of a computer-use agent: a text state (task, checkpoint, pending action), up to four images (screenshots and a crop), the questions of the NightWatch question bank asked at that checkpoint, and a probability label per answer.

No benchmark data. The desktop part comes from 360 tasks generated for this purpose. They share no task ids with OSWorld, and no instruction has a difflib ratio of 0.6 or more to any OSWorld instruction (the closest is 0.58). Each task starts from a freshly launched application and uses no OSWorld files, pages, or profiles. Only the OSWorld virtual machine is used as the environment. The task evaluator is infeasible: only the screens the agent passes through are used, never task success.

Contents

Folder Examples Source Labels
tasks/ 360 tasks generated instructions for the ten OSWorld applications (train_tasks.json lists them) –
desktop/ 16,205 agent runs on tasks/, three checkpoints (during reasoning, before dispatch, after execution) teacher decision model, gold labels from the frame stream where they are unambiguous
desktop/examples_nosettled.jsonl 16,205 the same examples without the settled question as above
settledness/ 7,316 the same desktop recordings hindsight: settled when the screen does not change again within 1 s
synthweb/ 50,040 procedurally generated web pages (dialogs, toasts, errors, loading, layout shifts, delayed enabling, ambient motion) exact ground truth

Splits are in the split field of every example (train / val, plus holdout / test in SynthWeb).

Training recipe of NightWatch-0.8B

  1. From vllm-sr/d3-lite: LoRA SFT on desktop/ plus up to 15,000 synthweb/ examples (soft cross-entropy + Brier).
  2. Then: LoRA SFT on settledness/ plus 4,000 examples of desktop/examples_nosettled.jsonl; temperature refit on the validation split.

The training code is in the NightWatch repository (training/).

Layout

Images are packed in tar shards. Extract every shard inside its folder to restore the paths the examples use:

for d in desktop settledness synthweb; do (cd $d && for t in images-*.tar; do tar xf "$t"; done); done

Example fields: id, split, task (desktop), domain, checkpoint, state, images (paths relative to the folder), questions, labels / soft (answer probabilities), label_source, meta.

Downloads last month
34

Collection including nullptr04/NightWatch-Data