Instructions to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S
# Run inference directly in the terminal:
llama cli -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S
# Run inference directly in the terminal:
llama cli -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S
# Run inference directly in the terminal:
./llama-cli -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Use Docker

docker model run hf.co/Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Llama new
LM Studio
Jan

vLLM

How to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Makdok/Flint-40B-A2.5B-24GB-128k-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Makdok/Flint-40B-A2.5B-24GB-128k-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Ollama
How to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with Ollama:
```
ollama run hf.co/Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S
```
Unsloth Desktop

How to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with Docker Model Runner:
```
docker model run hf.co/Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S
```

Lemonade

How to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Run and chat with the model

lemonade run user.Flint-40B-A2.5B-24GB-128k-GGUF-IQ3_S

List all available models

lemonade list

Hermes Agent

How to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Makdok/Flint-40B-A2.5B-24GB-128k-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Makdok/Flint-40B-A2.5B-24GB-128k-GGUF:IQ3_S" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Flint 40B-A2.5B - MixA GGUF for 24 GB GPUs at 128k context

A single mixed-precision GGUF of RecursiveSI/flint sized for one thing: running Flint fully on a 24 GB GPU with a 131,072-token context, weights and KV cache in VRAM, nothing offloaded to RAM.

File	Size	Notes
`Flint-40B-A2.5B-MixA-IQ4_XS-IQ3_S.gguf`	19.80 GiB (21.26 GB)	weights + llama.cpp-compatible chat template
`imatrix-flint.gguf`	450 MB	importance matrix used here (reusable)
`chat_template_llamacpp.jinja`	2 KB	same template as embedded, for `--chat-template-file`

This is an independent community quantization, not an official Recursive release.

TL;DR

Fits 24 GB at 128k with q8_0 KV: 22.4 GiB used on the card (RX 7900 XTX, Windows, Vulkan), about 1.4 GiB left for the desktop.
Lower KL divergence than the smaller quants that fit a 24 GB card: 26-31 % lower than unsloth UD-Q3_K_XL (17.2 GiB) on the same texts, see below.
Do not use a q4_0 KV cache with Flint. In my measurements it raised KLD by 67 % on code, more than all the extra expert precision buys. q8_0 KV costs about +6 %.
90 tok/s generation, ~3,300 tok/s prompt processing on an RX 7900 XTX (Vulkan).

Requirements

Flint support in llama.cpp is in PR #30928 (not merged when this was published). Build llama.cpp with that PR. If the PR changes the GGUF format before merging, this file may need a re-conversion.

The original Flint chat template uses a replace("\x00", "") filter that llama.cpp's Jinja engine cannot parse. The template embedded in this file has that single filter removed, nothing else. Tool calls (<tool_call>name<arg_key>...</arg_key><arg_value>...</arg_value></tool_call>) are parsed by llama-server into OpenAI tool_calls (checked on the PR build).

llama-server -m Flint-40B-A2.5B-MixA-IQ4_XS-IQ3_S.gguf --jinja \
  -c 131072 -ngl 99 -np 1 -fa on -ctk q8_0 -ctv q8_0 -b 2048 -ub 1024 \
  --temp 1 --top-p 0.95 --top-k 20 --min-p 0 \
  --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 3 --spec-ngram-mod-n-max 128

Keep -ctk q8_0 -ctv q8_0 (see the KV section).
Give Flint a large output budget: at 40,960 max tokens it was cut off on 3 hard AIME problems that it solved with 81,920. Recursive recommends at least 32,768, and their client configs use 65,536.
-ub 1024 makes prompt processing ~34 % faster than 512 for +0.2 GiB.
Flint has no MTP head; the n-gram drafter only helps on repeated text (code edits).

Recipe

Source: the official BF16 safetensors, converted with the PR's convert_hf_to_gguf.py. Base type Q8_0, then the routed experts are overridden:

llama-quantize --imatrix imatrix-flint.gguf \
  --tensor-type "blk\.([2-9]|[12]\d|3[0-6])\.ffn_(gate|up|down)_exps=iq4_xs" \
  --tensor-type "ffn_(gate|up|down)_exps=iq3_s" \
  flint-BF16.gguf Flint-40B-A2.5B-MixA-IQ4_XS-IQ3_S.gguf Q8_0

(llama-quantize applies the first --tensor-type that matches.)

Tensor group	Type
routed experts gate/up/down, layers 2-36	IQ4_XS
routed experts gate/up/down, layers 37-48	IQ3_S
attention (q, k, v, output), shared expert, dense FFN of layers 0-1, token_embd, output	Q8_0
router, norms, attention sinks	unchanged

The imatrix (153 chunks of 2,048 tokens) was computed on a Q8_0 of Flint over:

72 of Flint's own chat-formatted answers to coding and agent tasks (reasoning included), generated with its chat template;
300k characters of wikitext-2 train;
300k characters of Python/C source.

How the mix was chosen

Routed experts are 95 % of the weights, so the study changes one expert group at a time from a base with all experts in IQ3_S and everything else in Q8_0 (17.14 GiB), and ranks the changes by KLD reduction per GiB added (mean over 3 texts).

Change vs base	+GiB	KLD change wiki / code / traces	per GiB
down experts -> IQ4_XS (all layers)	+1.19	-21 % / -15 % / -18 %	~15 %/GiB
gate experts -> IQ4_XS	+1.19	-19 % / -16 % / -18 %	~15 %/GiB
up experts -> IQ4_XS	+1.19	-16 % / -17 % / -21 %	~15 %/GiB
down experts, layers 2-16	+0.38	-11 % / -6 % / -6 %	~20 %/GiB
down experts, layers 17-32	+0.41	-6 % / -3 % / -10 %	~15 %/GiB
down experts, layers 33-48	+0.41	-4 % / -5 % / -5 %	~11 %/GiB

Two findings drove the recipe:

gate, up and down experts are worth the same per GiB, so there is no reason to favour one;
earlier layers are worth more than later ones.

So MixA upgrades all three expert tensors in the first layers until the 24 GB / 128k budget is used up.

Quality: KL divergence vs Q8_0

llama-perplexity --kl-divergence, 2,048-token chunks, on 3 texts that the imatrix never saw:

wiki: wikitext-2 test, 8 chunks;
code: Python/C++ source, 8 chunks;
traces: 16 held-out Flint answers to different coding/agent prompts, 12 chunks.

Quant	Size (GiB)	KLD wiki	KLD code	KLD traces	top-1 same (code)
base (experts IQ3_S)	17.14	0.405	0.327	0.0570	84.5 %
unsloth UD-Q3_K_XL	17.23	0.328	0.298	0.0501	85.4 %
mix-b (IQ4_XS layers 2-25)	18.96	0.273	0.236	0.0425	86.9 %
MixA (this repo)	19.80	0.244	0.207	0.0344	87.6 %

Honest notes:

The reference is Q8_0, not BF16 (BF16 does not fit in 32 GB RAM + 24 GB VRAM for this measurement). Absolute values are not comparable with KLD-vs-BF16 tables published elsewhere, only the ranking within this table is.
Flint is much more sensitive to quantization than dense models of similar size (many small experts, 2.5B active parameters). Even at ~4 bpw on the experts, top-1 agreement on code is ~88 %.
The unsloth file is labelled "Flint Preview"; it was measured as published.

KV cache type matters a lot

Same model (MixA), same texts, only the KV cache type changes:

KV cache	KLD code	KLD traces	top-1 (code)
f16	0.207	0.0344	87.6 %
q8_0	0.220 (+6 %)	0.0345	87.8 %
K q8_0 / V q4_0	0.257 (+24 %)	0.0383	87.0 %
q4_0	0.346 (+67 %)	0.0549	84.6 %

Only 14 of Flint's 49 layers attend over the whole context (the others use a 513-token sliding window), so the KV cache is small anyway: about 1.95 GiB at 128k in q8_0. Saving 1 GiB with q4_0 is not worth it.

Tasks (sanity check, not a leaderboard)

12 AIME 2025 problems + 15 HumanEval+ problems, temperature 1.0, seed 42, RX 7900 XTX. Tasks cut at 40,960 tokens were re-run with 81,920. For reference, the same bench on my local dense 27B model (ThinkingCap-Qwen3.8-27B MixC, xhigh reasoning, MTP speculative decoding):

Model	Score	AIME	HumanEval+	generation	total wall time
Flint MixA (this repo)	23/27	11/12	12/15	90 tok/s	~4,530 s
ThinkingCap-Qwen3.8-27B MixC	25/27	12/12	13/15	67 tok/s	~5,320 s

With n=27 at temperature 1.0 a 2-task gap is close to noise. AIME is not what Flint was built for; it targets agentic coding (Terminal-Bench, SWE-bench), which this bench does not measure. Flint wrote ~10 % more tokens but finished ~15 % sooner, and reads long prompts ~6x faster than the dense 27B.

Speed and memory (RX 7900 XTX, Windows 11, Vulkan, AMD proprietary driver)

llama.cpp = PR #30928 + a small RDNA3 MMVQ fix (#29679), 128k context, 1 slot:

KV	`-ub`	generation (code)	prompt processing (12.6k tokens)	VRAM (server)	VRAM (whole card)
q8_0	1024	94.8 tok/s	3,310 tok/s	21.97 GiB	22.6 GiB
q8_0	512	92.9 tok/s	2,475 tok/s	21.77 GiB	22.4 GiB

Vulkan numerics were checked against CPU (perplexity within 1 %) before the study.

License

Apache 2.0, same as the base model. Flint is made by Recursive; see the original model card for training details, intended use and limitations.

Downloads last month: 568

GGUF

Model size

40B params

Architecture

flint

Hardware compatibility

3-bit

View +1 variant

Model tree for Makdok/Flint-40B-A2.5B-24GB-128k-GGUF

Base model

RecursiveSI/flint

Quantized

(81)

this model