miweru/Inkling-Small-REAP-137B-A12B-de-GGUF

Inkling-Small-REAP-137B-A12B-de (GGUF)

A REAP expert-pruned variant of thinkingmachines/Inkling-Small (276B-A12B → 137B total / 12B active), provided as GGUF quantizations.

Variants in this repo

Variant	Size	BPW	Notes
IQ4_XS (imatrix)	72.0 GB	~4.3	Recommended — importance-matrix-weighted, smaller and typically better than plain Q4_K_M
Q4_K_M	81.5 GB	4.84	Plain k-quant, no imatrix
imatrix.dat	217 MB	—	Importance matrix (538 entries) — build your own quants with llama-quantize --imatrix

The importance matrix was computed on a German-heavy DE/EN mix matching the pruning calibration (German multi-register human text, German model outputs, English code and math), so the quantization error budget follows the same language priorities as the expert pruning. Both variants are split into ≤45 GB parts; keep the parts of one variant in the same directory and load the first part.

The idea: produce a sparse model in the 100–140B class with a moderate quantization level (Q4_K_M, no aggressive sub-4-bit tricks), so that the quality loss budget is spent once on expert pruning and once on a well-understood quantization — not twice on extreme compression. We plan to spot-benchmark it against dense/sparse models of similar deployed size (e.g. Qwen3.5-122B-class models).

⚠️ Language disclaimer: the expert-pruning calibration was optimized for German and English (German-heavy multi-register calibration mix). Losses in other languages are presumably significantly higher — experts that were mostly used by other languages were preferentially removed. If you need multilingual coverage beyond DE/EN, use the original Inkling-Small.

What was done

• Method: REAP (Router-weighted Expert Activation Pruning, Lasby et al., ICLR 2026) — one-shot, no retraining. Per MoE layer, the 128 of 256 routed experts with the lowest router-weighted activation saliency were removed; router rows / e-score bias sliced accordingly. Shared experts, attention, dense layers, embeddings and MTP weights are untouched.
• Calibration: ~4M tokens, 56 % German (human-written text across 10 registers — news, legal, parliamentary, scientific, encyclopedic, spoken, easy language, essays, social, narrative — plus German instruction/chat data and German outputs of frontier models), 44 % English code / reasoning / tool-calling / chat.
• Quantization: llama.cpp Q4_K_M from BF16.

Calibration only collects routing statistics; no calibration text enters the weights.

Internal retention probes (not benchmarks)

Teacher-forced streaming probes against the BF16 base model (ΔNLL in nats, top-1 agreement with base). Spot checks, not standardized benchmarks:

Probe	ΔNLL vs. base	Top-1 agreement
German prose	+0.39	66.7 %
German factual text	−0.18	77.2 %
English prose	+0.72	73.0 %
Python	+0.68	81.0 %

For comparison: an English-only-calibrated pruning of the same ratio degraded German prose by +1.67 nats. Proper benchmark results (vs. Qwen3.5-122B-class models) are planned.

Usage

Requires a llama.cpp build with Inkling support — currently the add-inkling PR #25731 until it is merged:

git clone -b add-inkling https://github.com/danielhanchen/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bin/llama-cli -m Inkling-Small-REAP-137B-A12B-de-Q4_K_M-00001-of-00002.gguf \
  -p "Erkläre den Unterschied zwischen Bundesrat und Bundestag." -n 300

The split GGUF parts must sit in the same directory; llama.cpp loads them automatically from the first part.

Limitations

• DE/EN optimized — other languages likely degrade well beyond typical REAP losses (see disclaimer above).
• Code/tool-calling gives up a little quality relative to an English/code-only calibration (that trade-off was chosen deliberately).
• Text-only: vision/audio towers of the multimodal original are not part of this GGUF.
• No MTP speculative decoding in this GGUF (llama.cpp Inkling MTP support pending). The underlying pruned safetensors checkpoint retains the full MTP stack.
• Pruned models remain probabilistic compressions of the original — verify before production use.

About the base model: Inkling-Small

⚠️ Carry-over disclaimer: Everything in this section describes the original thinkingmachines/Inkling-Small (276B-A12B, BF16, multimodal). This repo contains a 50 %-expert-pruned, Q4_K_M-quantized, text-only derivative — the properties and scores below do not all carry over. Treat every number as an upper bound, not a claim about this model. Multimodal inputs are absent here entirely, and multilingual ability outside German/English was deliberately traded away during pruning calibration.

Key facts (original)

Property	Original Inkling-Small
Model type	Multimodal autoregressive transformer (text/image/audio → text)
Architecture	42-layer decoder, sparse MoE FFN: top-6 of 256 routed experts + 2 shared; hybrid local/global attention
Parameters	276B total / 12B active (this repo: 137B total / 12B active)
Context length	up to 1M tokens (long-context behavior of the pruned model is untested)
Numerics	BF16 and NVFP4 (this repo: Q4_K_M GGUF)
Languages	English, with general multilingual capabilities (this repo: DE/EN-calibrated)
Training data	Public, third-party-acquired and synthetic text/image/audio/video (see original card)
MTP	8 speculative multi-token-prediction layers (preserved in our safetensors, not in this GGUF)
License	Apache-2.0

Selected original benchmark scores

Scores of the unpruned BF16 original, from the official model card (where it is compared against Qwen3.5-397B-A17B, MiMo V2.5, Minimax M2.7 and closed models — see there for the full 10-model table). None of these numbers have been re-measured for this pruned+quantized variant.

Benchmark	Original Inkling-Small
SWEBench Verified	80.2 %
Terminal Bench 2.1 (best harness)	64.7 %
SciCode	48.7 %
GPQA Diamond	89.5 %
AIME 2026	95.5 %
HLE (text only / with tools)	31.6 % / 47.8 %
ARC-AGI-1 / ARC-AGI-2	84.0 % / 40.1 %
BrowseComp (with context mgmt)	77.4 %
IFBench	82.2 %
Global-MMLU-Lite	86.7 % (expect larger drops off DE/EN here)
SimpleQA Verified	20.6 %
StrongREJECT (safety)	98.4 %

Multimodal scores (MMMU Pro, Charxiv, MMAU, VoiceBench …) are omitted: the vision/audio towers are not part of this GGUF.

Deployment of the original

The original model has day-0 recipes for SGLang, vLLM, TokenSpeed, Unsloth and Hugging Face transformers (links in the original card), including MTP speculative decoding. This GGUF instead targets llama.cpp (see Usage above); our pruned safetensors checkpoint (TML layout, MTP intact, vLLM/SGLang-compatible) is not published here — open a discussion if you need it.

Safety, bias and limitations (inherited)

The original card documents safety training, known biases, risks and legal terms — all of that applies here at best unchanged and plausibly degraded: expert pruning is a lossy intervention whose effect on safety behavior and refusal calibration has not been separately evaluated. Review the original card's Safety / Bias / Legal sections before production use.

Provenance

• Base: thinkingmachines/Inkling-Small (Apache-2.0)
• Pruning: REAP streaming pipeline (calibration ran block-wise on a single RTX 5090 workstation without ever loading the 532 GB model into memory)
• Per-layer pruned expert lists: see reap_pruning.json in this repo

@inproceedings{lasby2026reap,
  title={{REAP} the Experts: Why Pruning Prevails for One-Shot MoE compression},
  author={Lasby, Lazarevich, Sinnadurai, Lie, Ioannou, Thangarasa},
  booktitle={ICLR}, year={2026}
}

---

Deutsch (Kurzfassung)

REAP-geprunte Variante von Inkling-Small: pro Layer wurden die 128 von 256 Experten mit der geringsten router-gewichteten Aktivierungs-Salienz entfernt (one-shot, kein Training), anschließend Q4_K_M-Quantisierung. Die Kalibrierung war bewusst deutschlastig (56 %, menschliche Texte über zehn Register plus deutsche Instruktionsdaten): deutsche Prosa verliert nur +0,39 nats gegenüber dem Original (statt +1,67 bei englischer Kalibrierung), deutscher Sachtext liegt auf Originalniveau. In anderen Sprachen als Deutsch und Englisch sind die Verluste mutmaßlich deutlich höher. Geplant sind Stichproben-Benchmarks gegen Modelle der Qwen3.5-122B-Klasse.