Maincode/Matilda-Platypus-13B-Base

Matilda Platypus 13B (base)

Any size you want. Matilda Platypus is a family of intermediate-size language models built by connecting the lower layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear Representation Bridge aligns the hidden representations at the connection. This checkpoint is the 13B member (cut 29+13): 13.51B parameters, 26.3 GFLOP per token, 91% of the larger parent's compute. It is a base model (no instruction tuning); see the technical report (technical_report.pdf) for the method and the full evaluation.

How this model is built

Component	Source	Layers	Notes
Lower block	Qwen/Qwen3-14B-Base (40 layers)	1–29	frozen at the released weights, loaded from the hub
Representation Bridge	learned	–	one linear map, 5120 → 4096 hidden units, 21.0M parameters (bridge.safetensors)
Upper block	Qwen/Qwen3-8B-Base (36 layers)	24–36 + final norm + output head	lightly adapted with low-rank updates (merged), shipped as upper_block/

The notation 29+13 means 29 layers of the larger parent followed by 13 layers of the smaller parent. The forward pass is: embedding and layers 1–29 of Qwen3-14B-Base → bridge → layers 24–36 of Qwen3-8B-Base → its final normalisation and output head. The tokenizer is the Qwen3 tokenizer.

Results

Base-model evaluation with lm-eval-harness 0.4.13, identical protocol for the model and both parents (0-shot multiple choice, MMLU 5-shot, GSM8K 8-shot chain-of-thought, BBH 3-shot chain-of-thought, HumanEval 0-shot and MBPP 3-shot with the generated code executed; HumanEval scored with markdown fences stripped). Scores are proportions.

Benchmark	Qwen3-8B-Base	Platypus 13B	Qwen3-14B-Base
HumanEval pass@1	0.677	0.750	0.720
GSM8K (8-shot CoT)	0.856	0.882	0.870
MMLU (5-shot)	0.768	0.796	0.805
BBH (3-shot CoT)	0.774	0.785	0.788
HellaSwag	0.786	0.796	0.814
ARC-Easy	0.798	0.794	0.819
ARC-Challenge	0.572	0.563	0.588
OpenBookQA	0.416	0.446	0.480
WinoGrande	0.723	0.754	0.747
PIQA	0.794	0.800	0.824
LAMBADA	0.711	0.722	0.742
MBPP pass@1	0.686	0.730	0.748
BoolQ	0.832	0.854	0.865

Above the smaller parent on 11 of 13 benchmarks and above the larger parent on 3 of 13, at 91% of the larger parent's compute (26.3 vs 28.8 GFLOP per token; the smaller parent is 15.7). Compute is the report's relative measure, normalised to the larger parent; it is not a verified latency or cost measurement. Full tables and figures: technical_report.pdf.

Training recipe (summary)

1. Initialise the bridge by least squares. Both released parents are run on a small calibration set and a linear map is fitted between their representations at the connection; initial language-modelling loss ranks candidate cuts.
2. Supervised adaptation. The bridge and light low-rank updates to the retained upper layers are trained on a few billion tokens of mixed web text, code, mathematics and worked solutions, with the lower block frozen at its released weights.
3. Reinforcement learning with verifiable rewards. A short policy-optimisation stage on code problems with unit tests and mathematics problems with checkable answers.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Maincode/Matilda-Platypus-13B-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda()
inputs = tok("The capital of Australia is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=32)
print(tok.decode(out[0], skip_special_tokens=True))

The remote code loads the lower block from Qwen/Qwen3-14B-Base, the upper block from upper_block/ and the bridge from bridge.safetensors, truncates both parents to the retained layers and runs them through their standard transformers forward passes with one KV cache per block. generate follows the standard transformers keyword arguments (max_new_tokens / max_length, min_new_tokens, stopping_criteria, logits_processor, do_sample, temperature, top_k, top_p, repetition_penalty, eos_token_id, pad_token_id, streamer), so the model runs unchanged under lm-eval-harness (--model hf --model_args pretrained=Maincode/Matilda-Platypus-13B-Base,trust_remote_code=True). Greedy and sampled decoding only; beam search raises NotImplementedError. Requires transformers >= 4.56 (tested on 4.56 and 5.x).

Remote-code revision 2 (2026-10-31): revision 1 ignored max_length and stopping_criteria and always produced 128 new tokens, which truncated chain-of-thought answers under lm-eval and under-reported generative benchmarks (GSM8K read ~0.58 instead of the table value); tie_weights() is now a no-op at the top level. Model weights are unchanged.

Limitations

A base model: it continues text and is not instruction-tuned or safety-tuned. Its knowledge and behaviour derive from the two Qwen3 parents and the adaptation data. The evaluation above is a base-model protocol shared by the model and its parents; differences of a few tenths of a point are within run-to-run variance.

Citation

Maincode Team (Yue Yang, Arsh Chawla, David Lemphers). Matilda Platypus: Any Size You Want. Intelligence You Need. Technical report, Maincode / Matilda Research, October 2026. Included in this repository as technical_report.pdf. Built on Qwen3 (Qwen Team, Qwen3 Technical Report, arXiv:2505.09388). Released under the Apache 2.0 license.