Jitesh K Prasad PRO

@dipankarsarkar You are right, the own-mix null is the correct one. We checked it on our side. SmolLM2-360M goes from +2.6 against its favourite letter to +0.9 against its own mix, and Gargi stays at about +0.2 either way. Against own mix only LFM2.5 (+13.3), the two Falcons (+8.2 and +7.9) and nltower (+3.9, we have its rows but cannot publish them) clear 2 points. So I think the own-mix gap is the column to show, not the share.

Rotation is not planned right now. But when we do it, yes, we will keep the unrotated column next to it, because like you said the gap between them is the bias.

@dipankarsarkar Thanks for actually running the parquet. We re-counted from the raw logs with the real answer key, and your numbers hold exactly. Gargi picks B on 94.1% of the questions and scores 24.89, against 24.65 for always answering B. Gargi-Base is 66.1% B, and accuracy on the non-B questions goes from 11.8 to 2.0 after SFT. So you are right, SFT did move something, it just pushed the model onto B.

I think the cleanest way to see it is each model against its own favourite letter. Only 5 of the 27 beat always answering their favourite letter by more than 2 points: LFM2.5-230M (+13.3), the two Falcon-H1-Tiny models (+9.5 and +8.5), nltower (+3.7) and SmolLM2-360M (+2.6). The other 22, including all of ours, are less than 2 points away from it. So for most of these models the letter score is basically just which letter they like.

On your question, I think rotating the options is the cleaner fix, because then a model stuck on one letter gets exactly 25 whichever letter it is. Off-modal accuracy puts each model on a different set of questions. For now we will add the favourite-letter share to the board, so anyone can see which letter each model leans on.

Would you also show each model's score minus its own favourite letter, or is the share enough?

@RakshithFury Yes, I think you are right about the thinking mode. At this size the gain is small. Gargi gets 32.9 on GSM8K with thinking off and 36.0 with it on, so about 3 points. On our smaller model Laghu (152M) it helped only on short arithmetic, and it actually made instruction following and multi-turn chat worse. We kept it mainly because I wanted the full package of a bigger model (thinking switch, multi-turn, tool format, 16K context) in something that runs on a laptop. But that is not where the quality comes from.

And I agree on RL also. Our Laghu try is one result with a weak base model, so it does not show that RL cannot work below 500M. If the base is good enough, it should help.

On your question: yes, basically. We are a small lab and we fund it ourselves, so a low budget is not really a choice for us. But I think it is also a useful question by itself, how far a small model can go if you are careful with the data and the evals. That is why we run every comparison ourselves and put the per-question results out.

@dipankarsarkar Following up on this. Because of your point we put the full table out properly, it is live now: https://huggingface.co/spaces/udgamlabs/small-model-board

It has 27 small models, all under 400M, all run the same way, with letter and cloze MMLU side by side and the cloze minus letter column like you suggested. The per-question data is also open if you want to check the dial yourself: https://huggingface.co/datasets/udgamlabs/small-model-board-scores. I have also credited you in the write-up: https://udgamlabs.com/blog/small-model-board

Two small corrections to what I said above. We re-checked everything from the raw logs. Falcon-H1-Tiny-90M-Instruct is -4.3, not -4.2. And nltower’s lead basically goes away once you score cloze by answer length, so it is really just LFM2.5 and the two Falcon models. Also the LFM2.5 we ran is their instruct model, not the base.

One funny thing we found: on letter MMLU, just answering “D” every time gets 26.9. A lot of these small models pick the same letter almost every time, so most letter scores under 27 are not saying anything about knowledge.

And to be fair, the OLMES paper (Gu et al., 2024) already showed small models need the cloze format. We just measured it on the current small models. Do tell me if anything looks off.

@RakshithFury Yes, basically two stages. First pretraining from scratch on about 17B tokens, then SFT on chat data and step-by-step maths solutions. The thinking mode also comes from the SFT, there is no RL in it.

And yes, RL is the obvious next step. We actually tried it on our smaller model Laghu (152M), GRPO with distillation from Gargi, and it did not work for us. On our GSM8K dev set it went from 8.6% to 8.2% instead of going up, and the model started looping more. So I think at this size the base model is the bottleneck, it does not get enough answers right for RL to learn much from.

If you have seen RL work well below 500M, do share, I would like to read it.

@dipankarsarkar Yes, I think the cloze column is the better one for the card. And thanks for doing the math on it.

On your question: no, our SFT did not move the dial, at least not in our runs. Gargi-Base is letter 25.3 / cloze 30.1, and Gargi (the SFT chat model) is letter 24.9 / cloze 29.6. So both moved down by about half a point, which is basically noise on 14k questions, and the gap stays at about 4.8. We see the same with SmolLM2-135M vs its Instruct version (letter 24.5 to 22.9, cloze 29.9 to 27.6) and with our older Udgam-384M, Base vs Chat (25.2 / 29.0 to 24.9 / 29.0). All of these are the same lm-eval setup, 0-shot, without the chat template.

I think the reason is that our SFT data is mostly chat and step-by-step maths, so there is probably not much A/B/C/D practice in it. With the chat template it may look different, we have not tried that.

One more thing that supports your dial idea. We ran about 25 small models on both, and the only ones where letter beats cloze are LFM2.5-230M (-8.3), Falcon-H1-Tiny-90M Base (-3.8) and Instruct (-4.2), and nltower-100m (-2.3). Falcon’s base model already has it, so for them the skill seems to come from pretraining, not from chat tuning.

@RakshithFury Good questions.

On RL: no, the Gargi you can download has no RL. It is SFT only, on chat data and step-by-step maths solutions, and the thinking mode also comes from SFT. So the 36.0 on GSM8K with thinking is from SFT alone.

On the needle numbers: no, it is not RULER. It is our own simple test, closest to RULER’s single-needle task. We hide one 5-digit code near the start of a long real document, ask for it at the end, and check if the model’s top prediction is exactly that code. It is 30 trials per length, so 93% at 16K means 28 of 30, and the error bars are wide. It has none of RULER’s harder variants like multiple keys or values, so it is an easier test and I would expect lower numbers on RULER.

@8BitStudio
Thanks for actually trying it out. This kind of feedback is really useful for us.

So first, a bit of context. We are a small and new lab, and we are still learning. We will make mistakes, like the MMLU one Dipankar caught above. We are not claiming Gargi is the best model of its size or anything like that. What we try to do is be honest in our reports, including where it fails.

And yes, what you found is real. Most of it is already in the card also. Gargi is often wrong on facts, arithmetic and code, and it is not a maths specialist. Instruction following is also weak. We measured IFEval at 26.4 (prompt-level strict), and even Falcon-H1-Tiny-90M-Instruct gets 60.8. So your 9 bullet points example is basically what we saw too.

But I do not think this is benchmaxing. The scores where Gargi is close to Qwen2.5-0.5B are ARC and WinoGrande, and both are multiple-choice. They check if the model can pick the right option, not if it can write a working script or follow an exact format. A 384M model trained on 17B tokens can do okay there and still be bad at code. That is basically what you ran into. Also, before training we removed anything that overlapped with the benchmarks we report, and we ran every model in the table ourselves on the same setup.

The sky example, if it is giving wrong answers on our own page, then it should not be there. Thanks for pointing it out.

So I think the honest summary is: decent on multiple-choice reasoning for the data it saw, but weak at code, maths, facts and following exact instructions. If you find other places where it breaks, do tell me. That helps us more than the scores.

You are right, thanks for pushing on this. I should not have called it chance and put it on the data based on the letter score alone.

We actually ran the cloze version (mmlu_continuation, lm-eval 0.4.13, 0-shot, same setup as the card) when we were building our next model. I just did not put it on the card. The numbers:

Gargi-Base: 30.1 (letter 25.3)
SmolLM2-360M: 33.8 (letter 25.5)
SmolLM2-135M: 29.9 (letter 24.5)
Gemma-3-270M: 28.5 (letter 26.2)

So yes, Gargi clears 25 by about 5 points, and SmolLM2-360M is ahead by about 3.7. That is a much smaller gap than the letter numbers make it look. I think the honest version is: there is a knowledge gap, but it is a few points, not twenty.

One more thing that supports your point: LFM2.5-230M scores 38.8 on the letter version but 30.5 on cloze, so the letter format really is a skill some models are trained for. Really useful catch, thanks.

Good question. No, we did not benchmax it. We ran every model in that table ourselves on the same setup (lm-eval, 0-shot, each model with its own start token), we did not copy numbers from the model cards. And before training we removed anything from our data that overlapped with the benchmarks we report, ARC, HellaSwag, WinoGrande, PIQA, MMLU and a few others. No benchmark training sets went into pretraining.

But I think the “almost equal” part needs some context. Gargi is close to Qwen2.5-0.5B only on ARC-Easy, ARC-Challenge and WinoGrande (62.8 vs 64.7, 32.3 vs 32.4, 54.6 vs 56.1). On world knowledge it is clearly behind. MMLU is 25.5 vs 47.3, which is basically chance for us, and HellaSwag and PIQA are also lower. That is expected, because knowledge comes from data and 17B tokens cannot hold what 18T can. We have mentioned this in the “where it falls short” section of the card also.

Also, small models tend to score close to each other on ARC and WinoGrande. SmolLM2-360M is ahead of both of us there. So the honest way to read it is: good at reasoning-style questions for the amount of data it saw, not overall equal to Qwen.

If you want to check it yourself, Gargi-Base is on HF. Just pass add_bos_token=True in lm-eval, otherwise the scores drop by a few points. Our contamination-free evals are open at github.com/udgamlabs/udgam-evals. Do tell me if you find something off!