verify-ppt/ppt-set_3b_v2
Updated
Verify PPT
community
AI & ML interests
None defined yet.
Recent Activity
Organization Card
Welcome to the verify-ppt project!
In this space, we release all the model checkpoints and pre-processed datasets used in the paper: Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior. For the codebase, please refer to our GitHub repository: https://github.com/gucci-j/verify-ppt-at-scale.
Model Checkpoints
500M
Click to expand the table for 500M models
| PPT Task | PT Data | Hugging Face Model Link |
|---|---|---|
| - | C4 | verify-ppt/c4_500m |
| k-Shuffle Dyck | C4 | verify-ppt/ppt_c4_500m_v2 |
| Control (C4) | C4 | verify-ppt/ppt-c4_c4_500m |
| - | Olmo3 | verify-ppt/olmo3_500m |
| k-Shuffle Dyck | Olmo3 | verify-ppt/ppt_olmo3_500m_v2 |
| Control (Olmo3) | Olmo3 | verify-ppt/ppt-olmo3_olmo3_500m |
| - | SmolLM3 | verify-ppt/smollm3_500m |
| k-Shuffle Dyck | SmolLM3 | verify-ppt/ppt_smollm3_500m_v2 |
| Control (SmolLM3) | SmolLM3 | verify-ppt/ppt-smollm3_smollm3_500m |
| - | Marin | verify-ppt/marin_500m |
| - | Marin (Seed 2) | verify-ppt/marin_500m_seed2 |
| - | Marin (Seed 3) | verify-ppt/marin_500m_seed3 |
| k-Shuffle Dyck | Marin | verify-ppt/ppt_marin_500m_v2 |
| k-Shuffle Dyck (Seed 2) | Marin (Seed 2) | verify-ppt/ppt_marin_500m_v2_seed2 |
| k-Shuffle Dyck (Seed 3) | Marin (Seed 3) | verify-ppt/ppt_marin_500m_v2_seed3 |
| Control (Marin) | Marin | verify-ppt/ppt-marin_marin_500m |
1B
Click to expand the table for 1B models
| PPT Task | PT Data | Hugging Face Model Link |
|---|---|---|
| - | C4 | verify-ppt/c4_1b |
| k-Shuffle Dyck | C4 | verify-ppt/ppt_c4_1b_v2 |
| Control (C4) | C4 | verify-ppt/ppt-c4_c4_1b_re |
| - | Olmo3 | verify-ppt/olmo3_1b |
| k-Shuffle Dyck | Olmo3 | verify-ppt/ppt_olmo3_1b_v2 |
| Control (Olmo3) | Olmo3 | verify-ppt/ppt-olmo3_olmo3_1b_re |
| - | SmolLM3 | verify-ppt/smollm3_1b |
| k-Shuffle Dyck | SmolLM3 | verify-ppt/ppt_smollm3_1b_v2 |
| Control (SmolLM3) | SmolLM3 | verify-ppt/ppt-smollm3_smollm3_1b_re |
| - | Marin | verify-ppt/marin_1b |
| - | Marin (Seed 2) | verify-ppt/marin_1b_seed2 |
| - | Marin (Seed 3) | verify-ppt/marin_1b_seed3 |
| k-Shuffle Dyck | Marin | verify-ppt/ppt_marin_1b_v2 |
| k-Shuffle Dyck (Seed 2) | Marin (Seed 2) | verify-ppt/ppt_marin_1b_v2_seed2 |
| k-Shuffle Dyck (Seed 3) | Marin (Seed 3) | verify-ppt/ppt_marin_1b_v2_seed3 |
| Control (Marin) | Marin | verify-ppt/ppt-marin_marin_1b_re |
3B
Click to expand the table for 3B models
7B
Click to expand the table for 7B models
| PPT Task | PT Data | Hugging Face Model Link |
|---|---|---|
| - | Marin (75.5B) | verify-ppt/marin_100b_7b |
| k-Shuffle Dyck | Marin (75.5B) | verify-ppt/ppt_marin_100b_7b |
Pre-pretrained Checkpoints
Checkpoints right after pre-pretraining (500 steps), before pre-training. Each repository stores the weights under the 500_hf/ subfolder, e.g. AutoModelForCausalLM.from_pretrained("verify-ppt/ppt_3b_v2", subfolder="500_hf"). The last column lists the pre-trained models above that were initialised from each checkpoint.
500M
Click to expand the table for 500M pre-pretrained models
| PPT Task | Hugging Face Model Link | Initialises |
|---|---|---|
| k-Shuffle Dyck | verify-ppt/ppt_500m_v2 | verify-ppt/ppt_c4_500m_v2 verify-ppt/ppt_olmo3_500m_v2 verify-ppt/ppt_smollm3_500m_v2 verify-ppt/ppt_marin_500m_v2 |
| k-Shuffle Dyck (Seed 2) | verify-ppt/ppt_500m_v2_seed2 | verify-ppt/ppt_marin_500m_v2_seed2 |
| k-Shuffle Dyck (Seed 3) | verify-ppt/ppt_500m_v2_seed3 | verify-ppt/ppt_marin_500m_v2_seed3 |
| Control (C4) | verify-ppt/ppt-c4_500m | verify-ppt/ppt-c4_c4_500m |
| Control (Olmo3) | verify-ppt/ppt-olmo3_500m | verify-ppt/ppt-olmo3_olmo3_500m |
| Control (SmolLM3) | verify-ppt/ppt-smollm3_500m | verify-ppt/ppt-smollm3_smollm3_500m |
| Control (Marin) | verify-ppt/ppt-marin_500m | verify-ppt/ppt-marin_marin_500m |
1B
Click to expand the table for 1B pre-pretrained models
| PPT Task | Hugging Face Model Link | Initialises |
|---|---|---|
| k-Shuffle Dyck | verify-ppt/ppt_1b_v2 | verify-ppt/ppt_c4_1b_v2 verify-ppt/ppt_olmo3_1b_v2 verify-ppt/ppt_smollm3_1b_v2 verify-ppt/ppt_marin_1b_v2 |
| k-Shuffle Dyck (Seed 2) | verify-ppt/ppt_1b_v2_seed2 | verify-ppt/ppt_marin_1b_v2_seed2 |
| k-Shuffle Dyck (Seed 3) | verify-ppt/ppt_1b_v2_seed3 | verify-ppt/ppt_marin_1b_v2_seed3 |
| Control (C4) | verify-ppt/ppt-c4_1b | verify-ppt/ppt-c4_c4_1b_re |
| Control (Olmo3) | Not available | verify-ppt/ppt-olmo3_olmo3_1b_re |
| Control (SmolLM3) | Not available | verify-ppt/ppt-smollm3_smollm3_1b_re |
| Control (Marin) | Not available | verify-ppt/ppt-marin_marin_1b_re |
3B
Click to expand the table for 3B pre-pretrained models
7B
Click to expand the table for 7B pre-pretrained models
| PPT Task | Hugging Face Model Link | Initialises |
|---|---|---|
| k-Shuffle Dyck | verify-ppt/ppt_7b_v2 | verify-ppt/ppt_marin_100b_7b |
Preprocessed Datasets
Pre-pretraining
Click to expand the table for pre-pretraining datasets
| Settings | Hugging Face repository |
|---|---|
| k-Shuffle Dyck | verify-ppt/ppt-shuff_dyck_v2 |
| Control (C4) | verify-ppt/c4-c4_baseline |
| Control (SmolLM3) | verify-ppt/smollm3-baseline_v3 |
| Control (Olmo3) | verify-ppt/olmo3-dolma3_mix_baseline_v2 |
| Control (Marin) | verify-ppt/marin-baseline_v3 |
| Set | verify-ppt/ppt-set_v2 |
| MP-Struct Core | verify-ppt/ppt-mpstructcore_v2 |
| NCA | verify-ppt/ppt-nca |
| Logic | verify-ppt/ppt-logic_mixed |
| Music | verify-ppt/ppt-music-blm-transpose |
| Math | verify-ppt/ppt-math-finemath |
| Code | verify-ppt/ppt-code-stack-v2-Python |
Pre-training
Click to expand the table for pre-training datasets
Citation
If you find these artifacts useful for your research, please consider citing the following paper:
@article{yamaguchi2026syntheticprepretrainingsurvivesscale,
title={Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior},
author={Atsuki Yamaguchi and Tatsuro Inaba and Joel Niklaus and Michal Štefánik and Aline Villavicencio and Nikolaos Aletras},
year={2026},
eprint={2609.39827},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2609.39827},
journal={arXiv preprint},
volume={arXiv:2609.39827}
}
models 124
verify-ppt/ppt-code_3b
Updated
verify-ppt/ppt-code_olmo3_3b
Updated
verify-ppt/ppt-code_smollm3_3b
Updated
verify-ppt/ppt-code_marin_3b
Updated
verify-ppt/ppt-c4_1b
Updated
verify-ppt/ppt-code_c4_3b
Updated
verify-ppt/marin_1b
Updated
verify-ppt/ppt-math_marin_3b
Updated
verify-ppt/ppt-math_olmo3_3b
Updated
datasets 148
verify-ppt/olmo3-dolma3_mix_baseline
Updated • 330
verify-ppt/smollm3-baseline_v2
Updated • 1.12k
verify-ppt/marin-baseline_v2
Updated • 538
verify-ppt/ppt-mpstructcore
Updated • 206
verify-ppt/ppt-set
Updated • 431
verify-ppt/ppt-shuff_dyck
Updated • 109
verify-ppt/ppt-shuff_dyck_default
Updated • 359
verify-ppt/ppt-code-stack-v2-Python
Viewer • Updated • 1 • 1.51k
verify-ppt/ppt-math-finemath
Viewer • Updated • 1 • 251
verify-ppt/ppt-music-blm-transpose
Updated • 1