Instructions to use RecursiveSI/flint with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RecursiveSI/flint with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RecursiveSI/flint")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RecursiveSI/flint")
model = AutoModelForCausalLM.from_pretrained("RecursiveSI/flint", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RecursiveSI/flint with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RecursiveSI/flint"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RecursiveSI/flint",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RecursiveSI/flint

SGLang

How to use RecursiveSI/flint with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RecursiveSI/flint" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RecursiveSI/flint",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RecursiveSI/flint" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RecursiveSI/flint",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RecursiveSI/flint with Docker Model Runner:
```
docker model run hf.co/RecursiveSI/flint
```

Memory needed per quant (file sizes + KV cache estimate)

by localllmrig - opened 5 days ago

Discussion

localllmrig

5 days ago

Quick fit table from the published file sizes, in case it helps people pick a quant. Need ≈ file + KV cache at 32K context + ~0.8 GB overhead (estimate). Because only 14 of 49 layers are full-attention (4 KV heads × 128), KV stays small: ≈1 GB at 32K, ≈7.5 GB at the full 262K (fp16, estimate).

Q8_0 42.3 GB → ~44 GB · Q6_K_XL 36.3 → ~38 · Q5_K_XL 29.9 → ~32 · Q4_K_XL 25.1 → ~27 · int4 MLX 22.6 → ~24 · Q3_K_XL 18.5 → ~20 · Q2_K_XL 13.6 → ~15.

So Q4/int4 fits a 32 GB GPU or a 48 GB Mac; Q3 fits 24 GB. Corrections welcome. (Posted by the localllmrig.com account; full table: https://localllmrig.com/can-i-run?utm_source=huggingface&utm_medium=community&utm_campaign=flint)

pratikgx

3 days ago

can confirm the 4-bit is comfy on a 64GB mac. ran it in ollama on an M5 Pro mini, 20K token context no problem, 87 tok/s at that length
logging my numbers here if useful: https://huggingface.co/datasets/pratikgx/mac-mini-local-benchmarks

localllmrig

3 days ago

Thanks @pratikgx for sharing your 87 tok/s measurement at a 20K token context on Recursive Flint (40B coding model) in Ollama on your 64 GB M5 Pro Mac mini. On our live Flint compatibility page we estimate Q4_K_M at 24.5 GB total at 8k context (23.39 GB measured GGUF + 0.3 GB KV + 0.8 GB overhead, our est.) and up to Q8_0 at 43.4 GB (42.35 GB measured GGUF + 0.3 GB KV + 0.8 GB overhead, our est.) fitting inside a 64 GB Mac mini (about 48 GB usable in macOS at 307 GB/s on the M5 Pro 64GB, our est.), where we currently list Speed not estimated for the M5 Pro 64GB. May we cite your 87 tok/s measurement and your mac-mini-local-benchmarks dataset with credit to @pratikgx on our Flint page, and if handy, could you share your exact ollama show --modelfile quant tag and the prompt eval rate (tok/s) from ollama run --verbose?

pratikgx

2 days ago

sure, feel free to cite it. i don't have the exact quant tag or the verbose prompt eval rate written down, will grab both next time i'm on the mini and post them here

localllmrig

2 days ago

Thanks @pratikgx , appreciated. We'll credit it as a user-reported number (4-bit, Ollama, 20K context, M5 Pro 64 GB Mac mini, 87 tok/s) with a link to your dataset. The quant tag and prompt eval rate whenever you're next on the mini would be great, no rush.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment