Model Submission Request: CLOD-149M (plz)

#240
by Murzik344 - opened

Repo: https://huggingface.co/Murzik344/CLOD-149M

  • 148,898,078 params
  • pretrained from scratch by me with the nanochat harness
  • open weights
  • Apache-2.0
  • safetensors on HF.
  • Weights are the genuine final training checkpoint (step 38,180), not a merge.

Recipe:

20B tokens - 18.2B ClimbMix-400B (single epoch) + ~10% terminal procedural-arithmetic anneal at LR floor; custom 32k digit-split BPE; one RTX 5050 Laptop GPU, 8 days.

Results (zero-shot, acc_norm; lm-eval-style likelihood scoring; ArithMark-3.0 official dataset):

HellaSwag 34.86
ARC-Easy 45.37
ARC-Challenge 26.88
PIQA 67.08
ArithMark-3 36.00
=> Int Index 19.64 by the published formula.

Reproducibility:

  • token_bytes.pt ships in the repo for byte-exact bpb checks
  • full nanochat config embedded in config.json.
Axiomic Labs org

Cool model, do you mind making a pull request on the leaderboard with the changes added?

Datdanboi25 changed discussion status to closed

Sign up or log in to comment