Dsg2/LS3-Micro-Dense-Base

This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-Micro-Dense. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.

• LS3-Micro-Dense-Base.bin: Weights 36.5 MB
• LS3-Micro-Dense-Base.bin.env: Training config: 355 B
• LS3-Micro-Dense-Base.bin.opt: Optimizer state: 36.4 MB
• LS3-Micro-Dense-Base.bin.opt.mu: Muon momentum: 36.4 MB
• LS3-Micro-Dense-Base.bin.opt.t1: Attention gate state 6.3 MB

LS3-Micro-Dense: First dense model in the LS family.

(acc norm except LAMBADA)	HellaSwag	ARC-E	ARC-C	PIQA	LAMBADA	ArithMark	*II	**Intelligence Density
LS2.5-108M-A17M-Base	28.24	36.66	23.21	57.40	17.89	37.00	9.89	0.231
LS-Micro-Base (11.49M)	27.41	34.72	21.93	54.84	11.06	35.50	7.24	0.909
LS3-Micro-Base (9.97M)	27.31	34.76	23.29	57.34	12.87	38.90	9.64	1.290
LS3-Micro-Dense-Base (9.89M)	28.53	39.27	20.31	56.58	16.15	47.50	11.99	1.212

With the help of a significantly bigger and curated dataset, LS3-Micro-Dense is able to achieve significant gains in LAMBADA, ARC-E and ArithMark. ARC-C remains below chance, while HellaSwag stays largely flat.

• Intelligence Index was calculated using the formula from Open SLM Leaderboard. (Results reproducable via Cudalm's built in bench tool)
• Intelligence density derived using the following formula: Intelligence Index / sqrt(Active Params (millions) * Total Params (millions))
• The model was benchmarked via Cudalm's LM eval tool.

---

Model Specs	#
Shape	Dense transformer
Hidden Dim	256
Layers	12
Attention heads	8
KV heads	2
FFN	660
Context length	2048
Sliding window	1024
Vocab	4096 (digits singular)
Optimiser	Muon 2D, AdamW else
Parameters	9,890,048 (FP32)

---

Training summary

	Stage A (stable)	Stage B (cooldown)
Steps	228,127	60,878 (total 289,005)
Tokens	14.951B	3.990B (total 18.94B)
Learning rate	1.05e-3 held, 1,000-step warmup	1.05e-3 falling to 2.1e-5
Time, speed	33 h, 128k tok/s	8.3 h, 133k tok/s

Datasets used:

Stage A (warmup and stable):

• HuggingFaceFW/fineweb-edu (8 shards, 7.2B tokens), ODC-By-1.0
• openbmb/Ultra-FineWeb, data/ultrafineweb_en (40 parts, 2.4B), ODC-By-1.0
• openbmb/Ultra-FineWeb-L3, qa (11 shards, 4B), Apache-2.0
• openbmb/Ultra-FineWeb-L3, multi_style (9 shards, 2.6B), Apache-2.0
• HuggingFaceTB/cosmopedia, data/web_samples_v2 (30 shards, 0.6B), Apache-2.0
• openbmb/Ultra-FineWeb-L3, multi_style (30 shards sifted from older corpus, 0.611B), ODC-By-1.0
• nvidia/OpenMathInstruct-2, refs/convert/parquet, default/train (8 shards, 1.5B), CC-BY-4.0
• HuggingFaceTB/finemath, finemath-4plus (40 shards, 1B), ODC-By-1.0
• ajibawa-2023/Education-College-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
• ajibawa-2023/Education-High-School-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
• ajibawa-2023/Education-Young-Children, refs/convert/parquet (2 shards, 0.2B), Apache-2.0
• HuggingFaceTB/cosmopedia: stanford, openstax, khanacademy (16 shards, 0.2B), Apache-2.0
• arithb: (self generated grade school level arithmetics, 0.9B), Self

Stage B (decay) on train splits:

• Rowan/hellaswag, default, train
• ARC-Easy, ARC-Challenge allenai/ai2_arc, ARC-Easy and ARC-Challenge, train
• github.com/ybisk/ybisk.github.io, master/piqa/data/train.jsonl and train-labels.lst
• allenai/openbookqa, additional, train
• allenai/sciq, default, train
• tau/commonsense_qa, default, train
• allenai/winogrande, winogrande_xl, train

---

Examples:

Temp 0.2

<|user|>
Do you like pineapple on pizza?
<|assistant|>
"Pineapple" is a pineapple-like fruit that is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C.
Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C.
Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is (truncated at 120 tokens)

Temp 0.7

<|user|>
How can I turn a chess piece into a stack of 100 dollar bills?
<|assistant|>
"Each dollar is divided into two parts. "
|Adding money to a stack of 100 dollars is $100+1 = 110$.
|Having a single stack of 100 dollars is $100+100 = 200$.
|Blending dollars is $110+100 = 210$.
|Glutening dollars is $100+100=200$.
|Bl (truncated at 120 tokens)

To try the model out yourself, benchmark it or fine tune it, clone Cudalm and build it with build.py (Blackwell 2.0 supported, or else build.py --cpu). Then download the LS3-Micro-Dense checkpoint in the files section and drop them into Cudalm's Models folder (folders will self init on first run).