SlayerLab/Slayer1B-1.03

Slayer1B 1.03

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. Slayer1B 1.03 is Slayer1B 1.02 trained further: training resumed from the final checkpoint of 1.02 (step 5720, constant learning rate) and continued to step 7150 (7 497 318 400 tokens in total, ≈ 7.0 tokens per parameter).

Like 1.02, 1.03 is a constant-learning-rate state: the final learning-rate decay is applied once, at the end of the whole Slayer1B line. Pre-training continues from this checkpoint on the rest of the prepared corpus.

• Previous versions: SlayerLab/Slayer1B-1.02, SlayerLab/Slayer1B-1.01, SlayerLab/Slayer1B-1.00
• Training checkpoints (with optimizer state) and run record: SlayerLab/Slayer1B-1.03-checkpoints
• Training curve (steps up to 7150; 6860–7150 from the L40S resume): https://track.fabryka.ai/run/7ca4f84f-3b07-4525-8cde-f3a936dc3425
• Benchmark series 1.00–1.03: https://track.fabryka.ai/run/1426c0a8-2f07-5a03-80fc-92f7e922aa4b

Results

lm-evaluation-harness 0.4.13, zero-shot, full test sets, fp32, batch 8, final checkpoint of each release. Same runner (runner_benchy, sha256 4f274a79…) for all columns; the 1.03 column was computed in a different software runtime (torch 2.8.0 / numpy 2.2.6 instead of torch 2.14.1 / numpy 2.5.3), so it is not an identical-environment comparison.

	1.00 (1430)	1.01 (2860)	1.02 (5720)	1.03 (7150)
training tokens	1.50 B	3.00 B	6.00 B	7.50 B
end of training	lr decay	lr decay	constant lr	constant lr
HellaSwag, acc_norm	0.2859	0.3080	0.3382	0.3476
PIQA, acc	0.5865	0.5903	0.6224	0.6257
LAMBADA (OpenAI), acc	0.1995	0.2639	0.3109	0.3150
LAMBADA (OpenAI), perplexity ↓	159.66	69.37	45.77	42.83
ARC-Challenge, acc_norm ¹	0.2142	0.2432	0.2500	0.2611
BoolQ, acc ¹	0.5343	0.6083	0.6055	0.6055
MMLU, acc ¹	0.2301	0.2308	0.2323	0.2292

¹ Benchmark caveat. A leak scan of the whole training superset (SlayerLab/slayer-pl-8x3b@d50df04a, 22 603 122 documents) found n-gram overlap with BoolQ, MMLU and ARC test items in the English shared part (shared/en: 1428 / 1429 / 1345 documents). Overlap is not a measured contamination rate; the assessment is in progress. 1.00–1.03 were trained on the unfiltered data; read these three rows with this caveat. MMLU is near chance level (0.25) for all releases.

Scores of single checkpoints differ: the intermediate 1.03 checkpoints 6900 / 7050 scored e.g. LAMBADA perplexity 40.78 / 45.68 and BoolQ 0.6180 / 0.5908 (all points in the benchmark series above); no cause is attributed. The leaderboard protocol used in the 1.00–1.02 cards (glint_parity_eval.py) has not been run on 1.03 yet.

Usage

The weights are a plain PyTorch state dict in safetensors; they are not a transformers AutoModel. The model class is GPT in train_gpt_ref_r23.py (included).

import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]

Model

Parameters	1 073 912 851 (input and output embeddings tied, counted once)
Architecture	decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128)
Blocks	pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases
Context	4096 tokens
Tokenizer	byte-level BPE, 32 768 tokens (tokenizer.json, sha256 0640d3bd…), EOS id 0
Weights	model.safetensors, fp32, sha256 86a7760c57d4f4243efdee8b128af337c30cc95fbfad4530d9fe0ec467f3e5fa, exported from ckpt_7150.pt (sha256 d7cad967…)

Training data

Source: SlayerLab/slayer-pl-8x3b, revision d50df04a6ad46876fa559a87cca5d3e2a96e7098. One stream of 4096-token windows, each window at most once; 1.03 continues the stream of 1.02:

steps	pack	train.bin sha256	tokens available	batches used
1–1801	pack-00	760ef0cc…	1 889 519 627	1801
1802–3594	pack-01	0494df58…	1 880 520 445	1793
3595–5389	pack-02	294a444d…	1 883 249 429	1795
5390–7150	pack-03	7d675734…	1 890 850 536	1761

Mixing, personal-data masking and pack construction as described in the 1.02 card. Personal data: rule-based masking (tags [EMAIL], [TELEFON], [PESEL], [NIP], [REGON], [DOWOD]) does not guarantee removal of every personal datum. The 5133 documents matching the leak scan are excluded only from pack-04 on (first filtered update: zero-based 7192, native step 7193), i.e. after 1.03.

Training

Hardware	steps 5721–6750: 4 × NVIDIA RTX 4090 (24 GB); steps 6751–7150: 4 × NVIDIA L40S (48 GB), resumed from ckpt_6750.pt (sha256 970518a1…) with the same trainer, recipe, data order and batch; DDP
Batch	256 × 4096 = 1 048 576 tokens per step (4 ranks × micro-batch 2 × gradient accumulation 32)
Steps	resumed at 5720 (1.02 ckpt_5720.pt, sha256 6b2a9b11…), ran 5721 → 7150
Tokens	7150 × 1 048 576 = 7 497 318 400 in total
Schedule	constant lr 6e-4 throughout
Optimizer	Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest
Final training loss	2.3955 (step 7150)
Resume check (L40S)	batch hash of zero-based iterations 6751 and 7149 (native steps 6752 and 7150) identical to the original run
Checkpoints	every 150 steps from 5850 and the final step 7150; all are constant-lr states
Trainer	train_gpt_ref_r24_1.py (sha256 881cb055…): the 1.02 trainer, releasing the checkpoint tensors from GPU memory after resuming; the model definition is unchanged (train_gpt_ref_r23.py in this repository)

Limitations

• 1 073 912 851 parameters trained on 7 497 318 400 tokens (about 7.0 tokens per parameter).
• Released without a final learning-rate decay (see above).
• Polish-dominant data; English is a minority of the mixture.
• Base model: no instruction tuning, no safety tuning.
• BoolQ / MMLU / ARC scores carry the leak-scan caveat above.

License

Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (shared/core_sa, shared/legal_sa; see the 1.00 card and the dataset card).