QLNI/shadow-assistant-data

SHADOW Assistant Data

Training data for SHADOW Assistant, a small assistant that runs offline on a normal laptop CPU. It reads, writes, does maths and code, looks at images, video and documents, keeps a memory on disk, and searches the web only when you say yes.

It is built around one country. Big models read the whole internet and try to know everything about everywhere. We want to see what a small model becomes when a large part of what it reads is about one place: Australia's English, its places, laws, history and everyday life, on top of a strong global base.

The model uses the frozen vocabulary from our paper Embeddings Are Overpriced?: one fixed code per token, never trained, so new words and new languages can be added later without retraining.

The text is complete: 260.9B tokens in 1,084 shards. Images and video are encoded: about 9.5B tokens. The tuning data (7.4M conversations) and the maths for the end of training (9.8B tokens, math/anneal/) are done; new Kimi conversations are still being added.

The model this data is for

part	what it is	size
Vocabulary	Gemma 4's 262,144 tokens, each with one frozen 512-bit code. Shared by input and output, never trained	16.8 MB
Body	about 135M parameters, ternary weights (BitNet b1.58) with int8 activations, 24 layers, width 640	about 34 MB
Eye	the EmbeddingGemma 2 vision tower in int8, for images, documents and video frames	about 218 MB
Engram	a 4B-entry n-gram memory at int4, read 8 rows per token straight from disk	about 2 GB on disk
Context window	12k tokens with full attention in every layer, keys and values at int4	about 78 MB RAM
Memory archive	up to 100M tokens of past conversations and documents, kept on disk as 1-bit keys and values, 1.7 KB per token	grows with use, about 173 GB at the full 100M, the user sets the cap
Index	a 512-bit signature for every 64-token block of the archive; for each question it finds the best blocks and loads about 2,048 tokens of them into the window	about 0.1 GB RAM for 100M tokens

How a question flows: pictures go through the eye and become tokens like words. The index searches the archive and brings the right past pages back into the window. The Engram adds stored facts. The body reads everything and answers.

In total: about 4.3B parameters (135M body, about 170M eye, 4B Engram entries, plus the frozen table), about 2.3 GB of files (16.8 MB table, 34 MB body, 218 MB eye, 2 GB Engram) plus the archive. Only the body, the Engram and the index are trained.

Speed, measured with the full design and random weights on a laptop CPU (Intel Xeon W-11955M, 8 cores):

short chat	531 tokens a second, first token after 146 ms
answering with the full 12k window in use	206 tokens a second (394 with a short sliding window in most layers)
reading text into the window	722 tokens a second
a photo or a document page	about 1 second
32 seconds of video, one frame a second	about 11 seconds
fetching 2,048 tokens from the archive	13.5 ms
peak RAM, mixed session filling the whole window	738 MB

Training grows the window in steps (2k, 4k, 6k, 8k, 10k, 12k) while the index learns to pick the blocks attention would want. Quality numbers come after training.

What is in it

260.9B text tokens, counted after tokenising, plus about 9.5B image and video tokens.

part	tokens	sources
Knowledge, global	107.6B	FineWeb-Edu 72.6B, DCLM-Baseline 35.0B
Knowledge, Australian	34.3B	every .au page in FineWeb-Edu 24.3B, .au pages in DCLM 6.0B, Open Australian Legal Corpus 2.06B, JoeyLLM Australian 1B 0.99B, Australian Wikipedia articles 0.37B, SpaceAI Australian Corpus 0.37B, House of Representatives Hansard 1998 to 2025 0.23B, Australian Dictionary of Biography 0.02B
Code	59.6B	OpenCoder FineWeb code corpus 52.0B, OpenCoder annealing corpus 7.5B
Maths	53.9B	Nemotron-CC-Math 4plus
Creative writing	3.6B	Cosmopedia stories 2.63B, Project Gutenberg books 0.91B, Gutenberg Poetry Corpus and public domain poetry 0.06B
Arithmetic	2.0B	generated, every answer checked: addition to long division, decimals, fractions, percentages, GST, units
Images	about 7.0B	FineVision 6.13B (which includes The Cauldron, Docmatix, DoclingMatix and olmOCR), SynthChartNet 0.43B, M4-Instruct 0.42B, Atlas of Living Australia 0.03B (256,874 photos, up to 5 for each species; more to come), CORD-v2
Video	about 2.5B	VISTA-400K 0.78B, LLaVA-Video-178K 0.69B, Vript 0.39B, Video-STaR (Kinetics-700 and Charades) 0.27B, FineVideo 0.26B, ShareGPT4Video 0.09B, ActivityNet Captions 0.05B, YouCook2, Australian Roads Dashcam
Maths for the anneal	9.8B	MegaMath Q&A 6.53B, MathCoder2 2.97B, TuluMath 0.25B, DolminoSynthMath 0.03B, CodeSearchNet (maths) and GSM8K train 0.003B, in math/anneal/

Tuning data, used after pretraining (sft/<set>/), about 7.4M conversations and 5.6B tokens:

part	sets
Code	OpenCoder SFT stage 1 (4.21M conversations, 2.57B tokens), OpenCoder SFT stage 2 (436k, 0.28B)
Maths	NuminaMath-CoT (859k, 0.44B), Orca-Math (200k, 0.07B), GSM8K train (7.5k)
Tools	ToolMind (368k, 0.97B), Toucan (115k, 0.34B), Glaive function calling (113k, 0.05B), When2Call (15k)
Conversation	SmolTalk (1.04M, 0.90B), used as it is, without conversations that name other models
Australian	Australian alignment set (11.8k), Open Australian Legal QA (2.1k), and new conversations written with Kimi (being added): everyday Australian life (30k), dates and the limits of what the model knows (2k), who SHADOW is (1.5k), working with files (2.5k), creative writing (3k)
Anything to JSON	JSON data extraction (484), plus the charts, receipts and documents in the vision data
Vision	image and video questions and answers, taken from the vision shards (FineVision, LLaVA-Video-178K, ShareGPT4Video, Video-STaR and others)

Format

• Tokens use the Gemma 4 vocabulary (262,144 tokens, the EmbeddingGemma 2 tokenizer).
• Each shard is raw tokens, 3 bytes per token, little-endian. Every document is <bos> text <eos>.
• Next to each shard: a .json with token and document counts and the source files, and a .counts.npz with how often each token appears. The unigram prior of the frozen table is the sum of these files.
• Layout: one folder per source inside each part, for example knowledge/fineweb_edu/, knowledge/legal/, code/opencoder_annealing/, creative/gutenberg/. Shards are named <source>_w<worker>_<number>.bin.

Images and video (vision/images/<source>/ and vision/video/<source>/, FineVision with one folder per subset, Australian images in vision/images/australia/<source>/):

• Each picture goes through the eye at its own size: photos up to 280 tokens, documents, charts and screens up to 1,120, video at one frame a second with up to 64 frames and up to 140 tokens a frame (a frame almost the same as the last kept one is left out; kept frames keep their real times). One token covers about 48 x 48 pixels.
• Each shard is a .npz with codes (the 512-bit code of every picture token, 64 bytes each, what the model reads), vec and vec_scale (the eye's vector for every token in int8, where the source licence allows it) and img (per picture: first token, token count, budget, width, height).
• Next to it, a .jsonl with one line per sample: the source file and row, which pictures belong to it, frame times for video, and the original text (questions, answers, captions), and a .counts.npz with the token counts of that text, the same as for the text shards.

Tuning (sft/<set>/): part_NNN.jsonl.zst, one conversation per line as {"messages": [...], "tools": [...], "source": ...} with roles system, user, assistant and tool (tool calls as {"name", "arguments"}), a part_NNN.counts.npz of token counts, and a summary.json per set.

Published here or rebuilt

Published in this repository, as tokens (and for images and video, as codes and captions, never the raw files):

• Knowledge: FineWeb-Edu, DCLM-Baseline, the .au pages of both, Open Australian Legal Corpus, SpaceAI Australian Corpus, JoeyLLM Australian 1B, House of Representatives Hansard 1998 to 2025
• Code: OpenCoder FineWeb code corpus, OpenCoder annealing corpus
• Creative writing: Project Gutenberg books, Gutenberg Poetry Corpus, public domain poetry, Cosmopedia stories
• Arithmetic: all of it (generated by us)
• Images and video: FineVision, SynthChartNet, M4-Instruct, CORD-v2, Atlas of Living Australia, VISTA-400K, FineVideo, ShareGPT4Video, Australian Roads Dashcam, each with its own licence
• Tuning: all the sets listed above, including everything we generated

Rebuilt from the original source with the scripts in rebuild/, not posted here:

• Nemotron-CC-Math (NVIDIA's terms allow model training only)
• Australian Wikipedia articles (CC BY-SA)
• Australian Dictionary of Biography (CC BY-NC-ND, Australian National University)
• Video sets for research use only, or with no licence stated: LLaVA-Video-178K, Vript, Video-STaR (Kinetics-700, Charades), ActivityNet Captions, YouCook2

Waiting for permission

These need permission for this use. We have asked, and each will be added if it is granted:

• Trove digitised newspapers, 1803 to 1954 (National Library of Australia)
• Senate Hansard, and House of Representatives Hansard before 1998 (Parliament of Australia)
• AustLII case law and legislation

Left out on purpose

• News sites and other sites whose terms forbid bulk downloading.
• Sets that only ship file IDs (Stack-Edu, GitHub file lists), and tuning sets written by models whose licences follow the trained model.
• MovieChat-1K (films with no licence) and MAmmoTH-VL (mostly repeats FineVision).

To try later

• State Library Victoria and State Library of NSW pictures (their image services need new access).
• FineDiving clips for Video-STaR, and the video part of M4-Instruct (the videos are not in its repository).

Licence

QLNI Non-Commercial Research Licence (see LICENSE.md): free for non-commercial research and education, with attribution. Each part also keeps the terms of its source.

Thanks

Compute support from CloudRift through their AI Builder Grant.

The House of Representatives Hansard comes from Lindsay Katz and Rohan Alexander's database of the Australian Parliamentary Debates (Zenodo 17351233, CC BY 4.0; Katz and Alexander, "Digitization of the Australian Parliamentary Debates, 1998-2022", Scientific Data, 2023), parsed from the transcripts published by the Parliament of Australia.

If you find this useful

Please mention QLNI and CloudRift when you use this data, for example: "Training data from SHADOW Assistant by QLNI (https://huggingface.co/datasets/QLNI/shadow-assistant-data), encoded with compute from CloudRift (https://cloudrift.ai)."

Contact

Sai Kiran Bathula, New South Wales, Australia. saikiranbathula@qlni.io