Dsg2/LS-Micro-Base

This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS-Micro. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.

• Model size: 28.1MB (FP32)
• Cudalm QLM optimiser: 26.1MB
• Ngrams container: 14.4MB
• T1 sidecar: 48KB
• Environment vars: 295B

LS-Micro: Smallest and fastest model in the LS family.

	HellaSwag	ARC-E	ARC-C	PIQA	LAMBADA	ArithMark	II*
LS-Micro-Base	27.41	34.72	21.93	54.84	11.06	35.50	7.24
LS-Tiny-Base	27.11	32.24	24.49	53.81	2.99	34.00	6.22

LS-Micro is 3x smaller than LS-TIny (11M vs 31M), and has 3x less active parameters (5M vs 15M), yet has a 16% higher intelligence index.

* Intelligence Index was calculated using the formula from Open SLM Leaderboard. (Results reproducable via Cudalm's built in bench tool, see bottom quickstart)

Pushing the limits of size to quality, LS-Micro-Base is a sparse 11.49M MoE model (A5.52M) pretrained on 5.7B tokens. Training at a speed of 255.6k tokens/s on a 5060ti 16GB, it offers an incredible training efficiency at ~2.9x faster tokens seen compared to LS2.5's 108M-A17M model. Fully pretrained in ~6 hours.

Model specs
Total parameters:	11.49M
Backbone:	7.28M
Active:	5.52M
Ngrams:	4.21M
Context length:	2048
Pretrain details
Tokens seen:	~5.7B
GPU time (5060ti):	22310s

*Note that it was trained with a 16k vocab, so 3.15M of the 5.52M active parameters are embedding.

Chatbench results:

Multiple choice:

category	n	acc	accnorm	chance
fact	25	48.0%	60.0%	25.0%
commonsense	20	65.0%	55.0%	25.0%
instruct	20	55.0%	50.0%	25.0%
relevance	20	5.0%	30.0%	25.0%
consistency	18	61.1%	72.2%	25.0%
arithmetic	15	46.7%	46.7%	25.0%
LS-Micro-Base	118	46.6%	52.5%	25.0%
LS-Tiny-Base	118	40.7%	47.5%	25.0%

Example: (Greedy)

<|user|>
Hi
<|assistant|>
I'm sorry, but I'm unable to assist with the problem. My current capabilities are limited to the functions provided to me. In this case, I can help you solve the problem.

The base model is horrendous at chat, and is intended to be used as a base to be finetuned for specialised tasks.

To try LS-Micro or to train/finetune the model, follow this quickstart guide:

1. Clone the latest version of Cudalm here (only windows supported currently, optimised for Blackwell 2.0/CPU).
2. Build with build.py/build.py --cpu'. Run cudalm to init all folders.
3. Download all LS-Micro files from LS-Micro-Base-Checkpoint\ and drop them in <cudalm.exe>\models\. Cudalm then automatically detects the model.

Voila, you can now start experimenting with the model. Some basic stuff you can try first:

• Check the model's config, and settings used to train it. > Model info
• Run the model: > Generate

Finetune guide:

To finetune the model, you first need to prepare some data.

Navigate to > Data tools.... You can download preconfigured mixtures of data, use files on your system or nitpick your own datasets.

• To use a preset corpus, navigate into > Data tools... > Download text.
• Otherwise, find a dataset on hugginface and navigate to > Data tools... > Browse Hugging Face, and pick out your chosen dataset from there. Note that datasets with gated access requires your HF_TOKEN to be set.

This guide will use the dataset MiniCPM5-2B-distill as an example. For your own text or other datasets, simply select the folder while tokenizing.

Navigate to > Data tools... > Browse Hugging Face, and search for "MiniCPM5-2B-distill". Cudalm automatically detects the format, so all you need to do is download it (~250MB).

Then you will need to tokenize the text so that the model can be trained on it. To do this, you will need a trained tokenizer. Luckily, you can extract an existing model's tokenizer. Use > Data tools... > Tokenizer from model and pull out LS-Micro-Base's tokenizer.

Now you can tokenize your data. Go to > Data tools... > Build corpus and set mode to "new" and give it a corpus name. For your own datasets or text, navigate to the folder the text files are in. Otherwise, select <cudalm.exe>\data\hf_MiniCPM5-2B-distill, then use the extracted tokenizer and select Run. It should take ~10s and produce <cudalm.exe>\data\<corpus name>

You're now ready to start fine tuning!

Go to > Train and select "LS-Micro-Base" as the model, and the tokenized corpus as the data directory (both auto-detected). Select Run to start training the model on the selected corpus. Progress prints and periodic saves will occur every few hundred iters.