HelloSun/flint_cpu_opt

flint CPU optimization (llama.cpp patch + two GGUF files)

Recursive Flint is a 39.8B-parameter mixture-of-experts coding model with about 2.5B active parameters per token (256 experts, top-6, 49 layers). This repo holds a llama.cpp CPU patch that speeds it up on Intel Xeon (AVX512-VNNI / AMX) and the two GGUF files it was tuned and measured on. The GGUF files are unmodified; every gain comes from code (patches/flint-cpu-opt.patch, base commit de5122d87). It is in the style of HelloSun/Qwen3.6-35B-A3B-GGUF_CPU_OPT.

file	source	quant	size
flint-UD-Q4_K_XL.gguf	unsloth/flint-GGUF	Q5_K 11.6 GiB + IQ4_XS 9.0 GiB + Q8_0 2.0 GiB (+ small Q4_K/Q6_K/IQ3_S)	23.3 GiB
flint-int4-q4_1.gguf	RecursiveSI/flint-int4-gguf	Q4_1 for every weight (token_embd and router F32)	24.2 GiB

Which file to use

Measured on Intel Xeon Platinum 8559C, 16 threads, with the patch (median of 3 interleaved rounds, tokens/s):

file	prefill pp256	prefill pp2048	decode tg128	PPL (wikitext-2, 32x512)
Q4_K_XL, stock llama.cpp	231	274	38.2	18.363 +- 0.795
Q4_K_XL, patched	296	307	42.8	18.277 +- 0.787
Q4_1, stock llama.cpp	183	189	48.3	18.260 +- 0.780
Q4_1, patched	266	297	55.7	18.240 +- 0.779

• Long prompts / large contexts (prefill-heavy): Q4_K_XL still prefills a little faster, but patched Q4_1 is now within ~10 % (it uses the tiled VNNI matmul too).
• Chat and agent loops with a cached prefix (decode-heavy): Q4_1. It reads fewer bytes per weight (0.625) and decodes ~30 % faster.
• Quality is equivalent within measurement error: the PPL differences above are smaller than the +-0.8 standard error of a 32-chunk test.
• Single runs on this shared host vary by about +-8 %; the numbers are medians, raw data in benchmarks/.

Gain of the patch (vs stock llama.cpp, same file)

file	pp256	pp2048	tg128
Q4_K_XL	1.28x	1.12x	1.12x
Q4_1	1.46x	1.57x	1.15x

For Q4_K_XL most of this comes from the Qwen3.6 patch kernels (Q4_K/Q5_K/Q6_K VNNI, AMX tweaks, MoE tile grouping) also applying to flint; against the Qwen3.6 patch alone the flint-specific part is small (pp256 +12 %, decode +2 %, pp2048 equal). The flint-specific changes:

1. IQ4_XS x Q8_K vec_dot in AVX512-VNNI (x86 had AVX2 only). Unsigned LUT (+128) with vpdpbusd, bias removed through bsums, L2 prefetch.
2. Q4_1 x Q8_1 vec_dot in AVX-512VL VNNI. Two blocks per iteration, packed into one ymm with memory-source vinserti128, low/high nibbles each through one vpdpbusd, fp16 headers converted together. A 512-bit version using cross-lane shuffles was slower than stock and dropped.
3. Q4_1 in the tiled VNNI GEMM for prefill. Q4_1 has no super-block scale, so the tiles gain per-32-block f32 scale/min arrays (sd/sm) and a Q41 kernel flag; activations reuse the existing Q8_1 vec_dot_type. Per 32-block: d_a * (d_w * dot(q, code) + m_w * sum(code)), integer dot/sum exact. This lifts pp256/pp2048 by ~34 %/42 % over the vec_dot path.
4. Q5_K 5th-bit merge with vptestmb + masked add (bit-identical to the previous version, no speed change).

Accuracy

Wikitext-2 test, 32 chunks x 512, KL divergence against the stated baseline logits of the same file:

comparison	KLD	same top-1	noise floor: stock vs stock with -fa off
Q4_K_XL, patched vs stock	0.0666 +- 0.0049	~92 %	0.0762 (also -ub 128: 0.0700)
Q4_1, vec_dot patch vs stock	0.0364 +- 0.0029	93.8 %	0.0385
Q4_1, tiled GEMM vs vec_dot patch	0.0372 +- 0.0026	94.2 %	0.0385

The KLD is inside the noise floor of stock itself: flint is very sensitive to tiny numeric differences (changing only the attention path of stock gives the same KLD), so it does not indicate a defect. Supporting checks: a graph-level mul_mat / mul_mat_id test gives the same relative L2 error versus a float reference as stock (Q4_K/Q5_K/Q6_K/IQ4_XS ~7e-3, Q4_1 ~5.4e-3, Q8_0 ~5.4e-3; the tiled Q4_1 path gives the same 5.29e-3/5.35e-3 as the Q4_1 vec_dot path). Integer dot products are exact; only float accumulation order changes. Why small differences get amplified was not verified (top-6-of-256 expert routing ties is a guess). The earlier version of this repo reported KLD 0.2058 for a different patch.

Where the time goes (why further kernel work is limited)

Measured DRAM read bandwidth on this host is 125-175 GB/s with 16 threads (single thread 15 GB/s). Decode reads about 2 GB of weights per token (~1.6 GB dense Q8_0 attention/shared-expert/lm_head, ~0.5 GB for the 6 selected experts), i.e. ~90 GB/s effective: mostly memory-bound. Per-type effective bandwidth during decode (Q4_K_XL): Q8_0 dense ~100-170 GB/s (lm_head ~170), IQ4_XS experts ~100, Q5_K experts ~50. The Q5_K gap is not kernel throughput (cache-resident Q5_K and IQ4_XS run the same, 232 vs 218 GB/s); a prefetch-all-experts experiment did not help and was dropped; the cause was not found. Prefill is compute-bound; once Q4_1 is in the tiled GEMM, FlashAttention and the tiled MoE matmul are the largest items (the per-row Q4_1 vec_dot remains only on the decode path). The Q4_1 tiled kernel carries per-32-block f32 scales/mins in the tiles (sd/sm) on top of the integer sub-block layout used by the K-quants.

Build and run

./build.sh        # clones llama.cpp at patches/BASE_COMMIT, applies patches/flint-cpu-opt.patch, builds with -DGGML_NATIVE=ON
taskset -c 0-15 llama.cpp/build/bin/llama-cli -m flint-UD-Q4_K_XL.gguf -t 16 -fa on -lm mlock

GGML_IQ4XS_PF, GGML_Q41_PF, GGML_K4_PF, GGML_K5_PF, GGML_Q6_PF tune the prefetch distances (defaults are the tuned values). patches/optional-profiler.patch adds GGML_PROF=1 per-op timing.

Limitations

• Validated only on Intel Xeon Platinum 8559C (AVX512-VNNI + AMX), 16 threads, on a shared host. Other CPUs are untested; the new kernels need AVX512-VNNI (the Q4_1 one also AVX512VL/F16C).
• The Q4_K_XL numbers were measured before the Q4_1 kernels were added; the K-quant code path is unchanged except for removing unused constants, and was not re-benchmarked (the Q4_1 tiled kernel was measured after).
• PPL and KLD use a short test (32 x 512 tokens); no downstream task evaluation was run.

Files

• patches/flint-cpu-opt.patch (base patches/BASE_COMMIT), patches/optional-profiler.patch, build.sh, machine.txt
• benchmarks/summary.md (Q4_K_XL), benchmarks/q41-summary.md (Q4_1), raw A/B CSVs (ab-interleaved-flint.csv, q41-ab-interleaved.csv, q41-tiled-ab-interleaved.csv), KLD logs (kld_vs_stock.log, q41-kld.log, q41-tiled-kld.log), op profiles, KLD bisect, graph_accuracy_test.cpp

Credits and license

Model: Recursive Flint (Apache-2.0), GGUF quantizations by Unsloth and RecursiveSI, unmodified. Patch builds on llama.cpp (MIT) and reuses ideas from the Qwen3.6 CPU-OPT patch. This repo is Apache-2.0 for the model files; the patch follows llama.cpp's license.