Flint-Q6K-IQ3XXS-GGUF

Q6_K dense layers + IQ3_XXS / IQ4_XS routed experts, 3.63 bits per weight.

A single GGUF of RecursiveSI/flint that fits the GPU working set of a 24 GB Apple Silicon Mac. The official INT4 builds (22.6 GB MLX, 26.0 GB GGUF) do not. On the measurements below it picks the same top token as a Q8_0 reference as often as the official INT4 GGUF does, with 31 % fewer bytes.

file	size	dense layers	routed experts
Flint-Q6K_IQ3XXS.gguf	18.0 GB (16.8 GiB)	Q6_K	IQ3_XXS gate/up, IQ4_XS down

sha256 efc33d5f57c7544cfc95c1a0eef58982d55a6745af39d701b788784207ddece4

Quality

KL divergence and perplexity against bartowski's Q8_0: llama-perplexity, 16 × 512-token chunks of English technical markdown, all files on the same CPU path as the reference.

file	size	PPL ratio vs Q8_0	mean KLD	same top token
RecursiveSI flint-int4-q4_1.gguf (official INT4)	26.0 GB	1.015	0.219	85.1 %
dense Q8_0 + these experts	~18.7 GB	1.034	0.226	86.8 %
this file (dense Q6_K)	18.0 GB	1.048	0.247	85.3 %
dense Q5_K + these experts	17.8 GB	1.072	0.284	83.1 %

This is not lossless. The 3-bit experts carry most of the cost, and its KLD is 13 % higher than the official INT4's. The official INT4 is derived from quantization-aware training, so its distance from the bf16-derived Q8_0 is not quantization error alone. Unlike Qwen3.5-35B-A3B, Flint does not take Q5_K dense layers for free, which is why this file uses Q6_K.

Speed

Apple M5 Pro, 24 GB, single stream, greedy, iogpu.wired_limit_mb=20480, measured right after a reboot.

engine	weights	128 tokens	512 tokens
tempo9 (engine-reported)	this file	98.2–102.0 tok/s	99.0–99.4 tok/s
llama.cpp llama-bench tg128 / tg512	this file	85.7 tok/s	85.1 tok/s
rec (patched for 24 GB, see below)	MLX INT4 dense + INT3 experts	150–159 tok/s	149–158 tok/s

rec is faster. It reads about 23 % fewer bytes per token (4-bit dense layers against Q6_K here), and its resident megakernel streams faster. It does not run on a 24 GB Mac as shipped: we removed its 48 GiB check, let it accept 3-bit routed experts, and raised the wired limit. At the default wired limit it hung the GPU.

Concurrent requests

rec serves one request at a time; tempo9 batches. Aggregate tok/s for 256-token greedy requests with distinct prompts, mean of 2 rounds:

engine	weights	1	2	4	8	16
rec	MLX INT4 dense + INT3 experts	159	145	134	138	141
tempo9	this file	100	154	228	274	337

At 16 concurrent requests the slowest one finishes in 12.2 s on tempo9 and 29.0 s on rec. The tempo9 numbers at 12+ concurrent requests need a build with the MoE batch fix (kzjeef/dash-infer#1258). Earlier builds drop to ~180 tok/s at 16.

Use

llama.cpp needs a build with the flint architecture (master edc3538 or later):

llama-server -hf thinkspread/Flint-Q6K-IQ3XXS-GGUF:Q6K_IQ3XXS -ngl 99

tempo9 needs a build with Flint support. As of 2026-10-23 that is not yet in a release.

tempo9 --hf thinkspread/Flint-Q6K-IQ3XXS-GGUF

Flint always reasons before answering (<think>…</think>) and ends a turn by opening the next one. Besides <|end_of_text|>, treat <|user|>, <|observation|> and <|developer|> as stop tokens, as the model's config.json does.

How it was made

Requantized from bartowski's Q8_0 with bartowski's importance matrix:

llama-quantize --allow-requantize --pure \
  --imatrix RecursiveSI_flint-imatrix.gguf \
  --tensor-type ffn_gate_exps=iq3_xxs --tensor-type ffn_up_exps=iq3_xxs \
  --tensor-type ffn_down_exps=iq4_xs \
  --output-tensor-type q6_K --token-embedding-type q6_K \
  RecursiveSI_flint-Q8_0.gguf Flint-Q6K_IQ3XXS.gguf Q6_K

Attention sinks, norms and the F32 MoE router are unchanged.

License

Apache 2.0, as published by RecursiveSI for Flint.