Abiray/flint-40B-A2.5B-Q4_K_M-GGUF

Flint (40B-A2.5B) - Q4_K_M GGUF

This repository provides the Q4_K_M GGUF quantization of RecursiveSI/flint

Flint is a state-of-the-art Mixture-of-Experts (MoE) model built specifically for coding agents, autonomous codebase exploration, bug fixing, and structured tool calling.

---

Model Specifications

Parameter	Specification
Developer	Recursive Superintelligence, Inc.
Architecture	Mixture-of-Experts (MoE) Decoder-only Transformer (flint)
Total Parameters	39.8 Billion
Active Parameters / Token	~2.5 Billion
Layers	49 total (2 dense + 47 MoE layers)
Experts	256 routed experts (top-6 active) + 1 shared expert
Context Length	Up to 262,144 tokens (hybrid sliding-window + global attention)
Quantization Type	Q4_K_M (4-bit k-quant medium)
Original Precision	BF16
License	Apache 2.0

---

Files Included

Filename	Quant Type	Size	Description
flint-40B-A2.5B-Q4_K_M.gguf	Q4_K_M	~24 GB	Standard medium 4-bit k-quantization offering optimal balance between perplexity and RAM/VRAM footprint.

---

Requirements & Compatibility

Important: Flint introduces a new architecture layout. Ensure your llama.cpp build includes support for the flint model architecture (PR #30928 or any release containing Flint support).

Hardware Recommendations

• RAM / VRAM: At least 28 GB – 32 GB of combined system RAM or VRAM to hold the model and KV cache during inference.
• CPU / GPU Offload: Thanks to the sparse MoE structure (only ~2.5B active parameters per token), token generation speeds remain fast even when partially or completely offloaded to CPU threads.

---

Usage

1. Running with llama-cli

# Direct CLI test
llama-cli \
  -m flint-40B-A2.5B-Q4_K_M.gguf \
  -p "<|im_start|>user\nWrite a Python script to find all palindromic substrings in a given string.<|im_end|>\n<|im_start|>assistant\n" \
  -n 1024 \
  -t 16 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20

2. OpenAI-Compatible Server (llama-server)

Start the local API server:

llama-server \
  -m flint-40B-A2.5B-Q4_K_M.gguf \
  -c 32768 \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 99

Query the server using the standard OpenAI client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="flint",
    messages=[
        {"role": "system", "content": "You are Flint, an expert coding agent."},
        {"role": "user", "content": "How do I optimize a PostgreSQL query using composite indexes?"}
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=4096
)

print(response.choices[0].message.content)

---

Recommended Generation Parameters

For optimal coding and agentic reasoning performance, use the upstream recommended sampling settings:

• Temperature: 1.0
• Top-P: 0.95
• Top-K: 20
• Reasoning Budget: For difficult problems or agent traces, allow at least 16384 to 32768 max tokens.

---

Acknowledgments & Credits

• Base model developed by Recursive Superintelligence, Inc.
• Quantized using llama.cpp