Post
2,864
We tested 27 small open language models, all under 400M parameters, in the same way. Same harness, same settings, and the model versions pinned. I think small models are really hard to compare from their own cards, so we put the whole thing out.
Board: udgamlabs/small-model-board
Dataset: udgamlabs/small-model-board-scores
Write-up on the Udgam Labs blog: https://udgamlabs.com/blog/small-model-board
The most surprising part for me was MMLU. In the usual A, B, C, D format, just answering "D" every time gets 26.9%, and 23 of these 27 models score at or below that. So at this size the letter format basically tests if a model can pick a letter, not what it knows. When we asked the same questions in cloze format, where the answer text itself is scored, all 23 did better.
Our own models are in there like everyone else. Laghu-Base is 5th of 19 models up to 200M, and Gargi-Base is 7.3 points behind SmolLM2-360M. So we are not on top, and that is fine. What I wanted was a fair comparison.
Do you compare small models on letter or cloze MMLU?
Board: udgamlabs/small-model-board
Dataset: udgamlabs/small-model-board-scores
Write-up on the Udgam Labs blog: https://udgamlabs.com/blog/small-model-board
The most surprising part for me was MMLU. In the usual A, B, C, D format, just answering "D" every time gets 26.9%, and 23 of these 27 models score at or below that. So at this size the letter format basically tests if a model can pick a letter, not what it knows. When we asked the same questions in cloze format, where the answer text itself is scored, all 23 did better.
Our own models are in there like everyone else. Laghu-Base is 5th of 19 models up to 200M, and Gargi-Base is 7.3 points behind SmolLM2-360M. So we are not on top, and that is fine. What I wanted was a fair comparison.
Do you compare small models on letter or cloze MMLU?