Instructions to use darioooooo0o/Spark-X2.5-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use darioooooo0o/Spark-X2.5-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use darioooooo0o/Spark-X2.5-4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "darioooooo0o/Spark-X2.5-4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "darioooooo0o/Spark-X2.5-4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Ollama
How to use darioooooo0o/Spark-X2.5-4B-GGUF with Ollama:
```
ollama run hf.co/darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use darioooooo0o/Spark-X2.5-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use darioooooo0o/Spark-X2.5-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M
```

Lemonade

How to use darioooooo0o/Spark-X2.5-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Spark-X2.5-4B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use darioooooo0o/Spark-X2.5-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use darioooooo0o/Spark-X2.5-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "darioooooo0o/Spark-X2.5-4B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Spark-X2.5-4B Community Quants (imatrix GGUF)

Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo

Community GGUF quantizations of XHToken/Spark-X2.5-4B, a 4.1B hybrid-attention (3:1 sliding-window:full) model with native 1M context. Quantized with an importance matrix calibrated on multilingual text, targeting fully GPU-resident inference on 4/8/12/16GB cards with at least 64K context (128K preferred).

Why the lineups look the way they do

Spark-X2.5-4B puts 69% of its parameters in FFN weights, and measured perplexity shows those FFN weights are uniformly sensitive: a layer-aware mix that drops ffn_up/ffn_gate to Q4_K while keeping everything else at Q6_K+ measured +0.76 PPL vs uniform Q6_K. Layer-aware mixing only pays off when the VRAM budget forces tensors below Q5_K - so the 12GB and 16GB quants are uniform (the headroom goes to context instead), while the 8GB and 4GB quants use mixes that protect the most sensitive tensors.

Available quants

file	card	size	BPW	allocation	max tested fit*
`Spark-X2.5-4B-Q6_K.gguf`	16GB	3.2 GB	6.56	uniform + imatrix	512K ctx (5.4 GB KV)
`Spark-X2.5-4B-Q5_K_M.gguf`	12GB	2.8 GB	5.78	uniform + imatrix	256K ctx (2.7 GB KV)
`Spark-X2.5-4B-Q3_K_M-8GB-mix.gguf`	8GB	2.4 GB	4.81	mix (below)	128K ctx (1.4 GB KV)
`Spark-X2.5-4B-Q4_K_M-4GB-mix.gguf`	4GB	2.0 GB	4.10	mix (below)	96K ctx (1.0 GB KV)

*fit = weights + q4_0 KV + ~1.1 GB CUDA/compute overhead, one slot, -ngl 99, flash attention on.

8GB mix allocation (Q3_K_M-8GB-mix)

tensor class	share of params	type
attn_qkv, attn_output	23%	Q6_K
token_embd, norms, gates	8%	Q8_0
ffn_down	23%	Q4_K
ffn_up, ffn_gate	46%	Q3_K

4GB mix allocation (Q4_K_M-4GB-mix)

tensor class	share of params	type
attn_qkv, attn_output	23%	Q4_K
token_embd	8%	Q5_K
ffn_down	23%	Q4_K
ffn_up, ffn_gate	46%	Q3_K

Measured quality (perplexity, held-out multilingual corpus, 512 ctx, CUDA, RTX 3060 12GB)

variant	size	PPL	delta vs Q8_0 (21.08)
Q8_0 reference	4.4 GB	21.08	-
Q6_K uniform (16GB)	3.4 GB	21.03	-0.05
Q5_K_M uniform (12GB)	3.0 GB	21.33	+0.26
8GB mix	2.4 GB	24.80	+3.73
4GB mix	2.1 GB	25.27	+4.19
uniform IQ3_XXS (rejected)	1.8 GB	32.62	+11.54

For comparison, rejected experiments: a 5.53 bpw mix (attn Q6_K / ffn_down Q5_K / ffn_up+gate Q4_K / embd Q8_0) measured 21.85 - worse than uniform Q6_K despite a smaller file. Any allocation that puts Q4_K or lower on ffn_up/ffn_gate while room remains for uniform Q5_K+ is a net loss on this model.

Context and KV cache

This is a hybrid model: only 9 of 36 layers carry full-attention KV (4 KV heads x 256 head dim). At q4_0 KV quantization that is ~10.1 KB/token:

64K: 0.68 GB | 128K: 1.36 GB | 256K: 2.72 GB | 512K: 5.44 GB
1M native context does NOT fit on any consumer card listed here (1M KV alone = 10.9 GB)

Recommended launch (llama.cpp)

Requires the vendor llama.cpp fork XHToken/llama.cpp (the spark2_5 architecture is not in upstream llama.cpp yet).

llama-server \
  -m Spark-X2.5-4B-Q4_K_M-4GB-mix.gguf \
  --chat-template-file chat_template.jinja \
  -c 131072 -ngl 99 -fa auto \
  -ctk q4_0 -ctv q4_0 -ctkd q4_0 -ctvd q4_0 \
  --temp 1.0 --top-p 0.95 --top-k -1 \
  --jinja --reasoning-format deepseek

Sampling per the model card: temperature 1.0, top_p 0.95, top_k -1. Thinking mode is on by default (qwen3-style reasoning); tool calls use the <tool_call>/<arg_key>/<arg_value> format and work through llama.cpp's auto tool-call parser.

Quantization method

Importance matrix computed with llama-imatrix on 200 chunks of multilingual text (6 MB corpus, 512-token chunks) from the Q8_0 GGUF.
Uniform quants: single Q6_K / Q5_K_M pass with --imatrix.
Mixes: per-tensor type assignment via --tensor-type-file (290 tensors). ffn_down is kept at higher precision than ffn_up/ffn_gate in every mix - it is the KL-sensitive FFN tensor.
Each variant validated with llama-perplexity on the same held-out corpus, sequentially on one RTX 3060 12GB, VRAM freed between runs.

Speed (RTX 3060 12GB, llama-bench, p512/n128, -ngl 99, -fa on, f16 KV, 3 reps)

variant	prefill tok/s	decode tok/s	TTFT ms	peak VRAM
Q6_K	3161.6	77.9	174.8	4.58 GB
Q5_K_M	3376.1	89.2	162.9	3.97 GB
8GB mix	~3300	~88	~163	~3.7 GB
4GB mix	3346.7	84.7	164.8	3.36 GB

At 40-60K context decode drops to 55-78 tok/s (long-context penalty); prefill holds 2500-3200 tok/s.

Agentic tool-use reliability (HermesBench subset, RTX 3060 12GB, 64K ctx)

Each quant ran a 6-task agentic probe subset (file/terminal, memory, todo, code_execution, session_search, cronjob tool classes) through a real agent harness, graded on both task correctness AND use of the required tool classes. Zero false-dones and zero timeouts on all quants. One harness-broken task excluded (failed identically on every quant including near-lossless Q6_K).

Tool-requirement compliance is stochastic at temp 1.0: repeated runs of the same task on the same quant flip between using and skipping the required specialized tool. Across 3 runs of 2 decisive tasks per quant:

quant	requirement-miss rate	runs
Q6_K	1/6 (17%)	3x (task-001, task-004)
Q5_K_M	1/6 (17%)	3x
Q3_K_M-8GB-mix	4/6 (67%)	3x
Q4_K_M-4GB-mix	3/6 (50%)	3x (one run lost to a harness timeout)

Key observations:

Skipping the required tool is run-to-run variance for every quant, not a hard defect - even near-lossless Q6_K missed once. The mixes miss more often (50-67% vs 17%), so the quant does shift the odds, but any single run's pass/fail is not a reliable signal.
Raw task competence is intact everywhere: in every miss on every quant, the task was still completed correctly (right answer, right file) using generic tools instead of the required specialized one. Quant damage at these bitrates degrades tool-selection discipline, not capability.
Practical guidance: if your agent harness enforces tool-class requirements strictly, expect the mixes to fail such checks 3-4x more often than the uniform quants. If you only care about task outcomes, all four quants complete the work.

Files

file	size
Spark-X2.5-4B-Q6_K.gguf	3.2 GB
Spark-X2.5-4B-Q5_K_M.gguf	2.8 GB
Spark-X2.5-4B-Q3_K_M-8GB-mix.gguf	2.4 GB
Spark-X2.5-4B-Q4_K_M-4GB-mix.gguf	2.0 GB

License

Apache 2.0, matching the base model. Quantizations are derivative works of XHToken/Spark-X2.5-4B.

Downloads last month: 216

GGUF

Model size

4B params

Architecture

spark2_5

Hardware compatibility

3-bit

4-bit

5-bit

6-bit

Model tree for darioooooo0o/Spark-X2.5-4B-GGUF

Base model

XHToken/Spark-X2.5-4B-Base

Finetuned

XHToken/Spark-X2.5-4B

Quantized

(55)

this model