chrome list | gimp list | libreoffice_calc list | libreoffice_impress list | libreoffice_writer list | multi_apps list | os list | thunderbird list | vlc list | vs_code list |
|---|---|---|---|---|---|---|---|---|---|
[
"0cd34de7-c200-43f6-bbdc-54e4f29173e6",
"139f088a-6807-4a85-868a-d3916e4fb4ab",
"17f11329-1d1b-456f-9e6e-acec9752960c",
"1e987a1d-804c-4a57-a877-c89e14b96172",
"27308370-ac79-4fd1-a979-75f846851e87",
"2acf70e9-5ce7-4500-873f-2f850b100208",
"33497f7f-2548-4ebf-92f1-a1d3abb6f63a",
"3ca6c72f-7549-4618-98... | [
"003f4ab4-721c-4135-a3b8-7c05b28e8c53",
"041b86a3-1a5f-41ac-bc59-6c380f41d4a4",
"0686c020-5bf5-43eb-9201-051fef806bf7",
"179c3e5e-f1e5-43f5-b2ae-e59d626a6b90",
"23b5e153-9518-48e3-9148-fc5a1a0ab63d",
"2cb07954-8220-4010-923b-25d32264f527",
"366aa110-c775-4879-af1b-5b1ee595cfa2",
"38623924-94d4-4029-bd... | [
"03b56137-8823-4e2a-95b1-753ec589d7b9",
"05b0c883-625d-4fec-9d74-2012c93ca9b0",
"12949b33-c561-4d2a-953e-e7961755c333",
"1a4235a4-a06a-4264-9eb9-e3a0c2b1f1ff",
"21b544f3-7541-4fa8-b397-7deb2d3e836e",
"21f8076e-b62b-4aeb-9bbc-a99acb4b5b11",
"22da0544-70d2-44fe-bd9d-282520bb5f0a",
"2bad1da1-e981-47a6-98... | [
"04de74cf-e3b1-43d5-b219-fd6f0bbffd1b",
"04e1cb16-2127-4f2d-a6c5-8cc08aaa99e6",
"0a0eb6f2-3b2b-48be-b135-b68c62539efe",
"0a42ca7d-e173-4302-add1-598b8c0ac7d4",
"123c6cd1-42d0-48ae-8a7e-14dad55c2545",
"1fb9805b-3b8f-4d5f-8480-79641da3a81b",
"2a06f4e5-62e5-4a5c-9631-f510220d6e01",
"2dab3dde-96de-4f7d-ab... | [
"00e29c5a-1fbb-4aae-86b4-84186e7b6714",
"0d4d369c-3bdd-4497-9218-2204c0d8e0c2",
"0ea5e24e-35e8-46b3-8c2f-c77b365983b7",
"1f398f9b-2bbd-4b81-ab36-8c7becdd8390",
"2657a92b-35f2-49f7-bca8-2b1e185b03f2",
"269bf124-ad55-4714-90dc-04889b9a8832",
"27993ee3-47ee-47ee-b434-e1ed97fbe4e9",
"377c7d07-a0d2-4e74-93... | [
"05c5e0b4-6b1a-4a72-a64d-e954a51a42c7",
"1148ac53-10af-468d-977d-54e04aadd697",
"11ae0817-a112-487b-9d9d-4415cccedc36",
"13dcab6a-1220-4a0e-9259-aa4fac199123",
"1ac9f041-ca0a-4d9d-97de-55313edae417",
"1feb94fc-5a29-4005-94fe-58116d7dd8c0",
"21a0f463-b842-4585-a277-7a8e7711d893",
"3064b551-9283-41a0-9d... | [
"04f701f9-2c41-4886-98b2-edeb135c83b2",
"08e9f469-dc86-4936-8c9f-fb087d4adf80",
"0c76f872-f527-48c8-b99c-a2d5bed0650d",
"0dcc75dd-7a9a-4421-a2e0-f404337c1b42",
"0df8972f-d1ee-4973-8d64-2a008f519eb5",
"1730da0e-53e5-4b91-8fae-59ffd566de37",
"1d766da2-da2c-47cd-a81c-115011616c49",
"1ff3bf87-79ea-4c2f-8b... | [
"1465f0c5-c304-4dfc-adfc-0a123b0e0982",
"180ba527-7988-43c0-993c-6b1f300c390e",
"2254590f-1ee4-461c-a077-4c979bae9c66",
"3a8feb24-527f-4b26-a313-3e77e4e7d0ab",
"4fb3c4a6-0f7d-4d79-9e72-aeda323af43f",
"5b3f71e0-fbc4-475a-a7fa-94c9bbf7094a",
"760bb839-2f54-4921-92a1-574edf4543ba",
"7f217aea-211c-4c33-a5... | [
"0f77e8ca-e796-44e6-b56e-f0ffbb4e1b36",
"1752e7c8-4402-4865-b59a-c1d690c28f28",
"23315cbb-d5ce-4931-96c4-90e1d4aedfbc",
"2ba4029c-be74-4bdb-8acf-0a090fe67c18",
"33ef23db-84bc-461d-a5bd-11e18f04788b",
"4140c1a7-1e82-4bea-9622-4a9171420df7",
"49f54f4d-5729-4d32-af8c-5bc7abc829c5",
"68c73326-46a9-490b-b3... | [
"033e9f78-776d-4a45-9dfd-bd433547b79b",
"07d26a90-ea78-48b9-ad0d-9f6b38e2393c",
"08446086-f702-45df-8f19-13fcd11ff8c7",
"0c19e825-0b15-4147-8ff4-9e5c34f4586c",
"1fce5281-3c90-44b4-b8c1-f8e812e01601",
"23d08e4c-e5dc-460e-9a85-a01cb6acf79d",
"2938f1a6-b54f-4386-81d1-ea347cebfb23",
"2b071642-1394-4cd6-b4... |

NightWatch training data
Training data of NightWatch-0.8B, the watcher of NightWatch (Don't Sleep, Keep Watch: Proactive Perception for Computer-Use Agents). Each example is one decision point of a computer-use agent: a text state (task, checkpoint, pending action), up to four images (screenshots and a crop), the questions of the NightWatch question bank asked at that checkpoint, and a probability label per answer.
No benchmark data. The desktop part comes from 360 tasks generated for this purpose. They share no task ids with
OSWorld, and no instruction has a difflib ratio of 0.6 or more to any OSWorld instruction (the closest is 0.58). Each
task starts from a freshly launched application and uses no OSWorld files, pages, or profiles. Only the OSWorld
virtual machine is used as the environment. The task evaluator is infeasible: only the screens the agent passes
through are used, never task success.
Contents
| Folder | Examples | Source | Labels |
|---|---|---|---|
tasks/ |
360 tasks | generated instructions for the ten OSWorld applications (train_tasks.json lists them) |
– |
desktop/ |
16,205 | agent runs on tasks/, three checkpoints (during reasoning, before dispatch, after execution) |
teacher decision model, gold labels from the frame stream where they are unambiguous |
desktop/examples_nosettled.jsonl |
16,205 | the same examples without the settled question |
as above |
settledness/ |
7,316 | the same desktop recordings | hindsight: settled when the screen does not change again within 1 s |
synthweb/ |
50,040 | procedurally generated web pages (dialogs, toasts, errors, loading, layout shifts, delayed enabling, ambient motion) | exact ground truth |
Splits are in the split field of every example (train / val, plus holdout / test in SynthWeb).
Training recipe of NightWatch-0.8B
- From vllm-sr/d3-lite: LoRA SFT on
desktop/plus up to 15,000synthweb/examples (soft cross-entropy + Brier). - Then: LoRA SFT on
settledness/plus 4,000 examples ofdesktop/examples_nosettled.jsonl; temperature refit on the validation split.
The training code is in the NightWatch repository (training/).
Layout
Images are packed in tar shards. Extract every shard inside its folder to restore the paths the examples use:
for d in desktop settledness synthweb; do (cd $d && for t in images-*.tar; do tar xf "$t"; done); done
Example fields: id, split, task (desktop), domain, checkpoint, state, images (paths relative to the
folder), questions, labels / soft (answer probabilities), label_source, meta.
- Downloads last month
- 34