Instructions to use hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming")
model = AutoModelForCausalLM.from_pretrained("hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming

SGLang

How to use hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming with Docker Model Runner:
```
docker model run hf.co/hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming
```

LFM2.5-24B-A2B-RCO-4.0-streaming

Mixed-precision MLX quantization of LiquidAI/LFM2.5-24B-A2B using RCO (Rate-Distortion Optimization) allocation with Hessian-weighted error database, built via streaming/layer-by-layer error_db generation.

Model

Base: LiquidAI/LFM2.5-24B-A2B
Parameters: 24B total, 2.45B active
Architecture: LFM2.5-MoE (40 layers: 30 conv + 10 full-attention, 64 experts top-4)
Context: 131,072 tokens
Quantization: RCO mixed-precision affine, group_size=64, target 4.0 bpw

Usage

from mlx_lm import load

model, tokenizer = load("hf://hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming")

Performance

Metric	Value
Model size	13.00 GB
Bits per weight (target)	4.0
Bits per weight (actual)	mixed (4–6 bpw)
Mode	affine
PPL (wikitext2)	13.262 (first 2048 tokens)
Throughput	36.4 tok/s
Degradation vs BF16	+19.51%

Streaming Error DB

This quant was built using the streaming/layer-by-layer error_db approach (build_error_db_streaming.py). The error_db was built layer-by-layer from BF16 shards:

Phase 1: Load BF16 model, wrap linear layers to capture activations, run calibration (50 × 512 tokens from Pride & Prejudice)
Phase 2: Load BF16 weights one tensor at a time from safetensors shards, measure Hessian-weighted quant error at bitwidths {2,3,4,6,8}
RCO knapsack allocation under 4.0 bpw budget
MLX affine quantization with expert stacking (switch_mlp 3D tensors)
Validation on wikitext2 test set (first 2048 tokens)

A/B Comparison (4-bit RCO vs RTN)

Model	PPL	TPS	Notes
RCO 4.0 (streaming)	13.262	36.4	This model
RCO 4.0 (non-streaming)	13.262	36.1	`hermitdave/LFM2.5-24B-A2B-RCO-4.0`
RTN 4.0 (uniform)	13.83	36.4	`hermitdave/LFM2.5-24B-A2B-RTN-4.0`

Key finding: Streaming and non-streaming error_db produce identical PPL (13.262), confirming the layer-by-layer approach is correct.

Quantization Details

This quant was produced with the rco-mlx pipeline:

Layer-by-layer introspection of the BF16 checkpoint
Hessian-weighted error database built from calibration text
RCO knapsack allocation under the 4.0 bpw budget
MLX affine quantization with expert stacking (switch_mlp 3D tensors)
Validation on wikitext2 test set

License

This model inherits the license of the base model. See LiquidAI/LFM2.5-24B-A2B for details.

Downloads last month: -

Model tree for hermitdave/LFM2.5-24B-A2B-RCO-4.0-streaming

Base model

LiquidAI/LFM2.5-24B-A2B-Base

Finetuned

LiquidAI/LFM2.5-24B-A2B

Quantized

(84)

this model