Hub documentation
Local Agents with llama.cpp
Local Agents with llama.cpp
You can run a coding agent locally, entirely on your own hardware. Several open-source agents can connect to a local llama.cpp server to give you an experience similar to Claude Code or Codex β but everything runs on your machine.
How It Works
The setup has two components running locally:
- llama.cpp server β Serves the model as an OpenAI-compatible API on
localhost. - Your agent β Sends prompts to the local server, reasons about tasks, and executes actions.
βββββββββββ API calls βββββββββββββββββββββ
β Agent β ββββββββββββββββΆ β llama.cpp server β
β β ββββββββββββββββ β (local model) β
βββββββββββ responses βββββββββββββββββββββ
β
βΌ
Your files,
terminal, etc.Set up llama.cpp server
1. Set Your Local Hardware
Set your local hardware so it can show you which models are compatible with your setup.
Go to huggingface.co/settings/hardware and configure your local hardware profile. Then select llama.cpp in your Local Apps settings, as this will be the engine youβll use.
2. Find a Compatible Model
Browse for GGUF models compatible with Llama. Once selected, download the model on your hardware.
If you need inspiration, the Llama app will suggest a few models adapted to your hardware device (see Step 3).
3. Install the Llama App
You can easily install llama.cpp on your machine:
- Mac and Windows: install the Llama app, a minimalist desktop app that sits on your system tray
After the application is installed follow the steps to start using your favourite model, recommended for your system.

- Linux (or if you only want the command line): one-line install command:
curl -LsSf https://llama.app/install.sh | sh
Verify the installation:
llama --version
4. Start the router
If you installed the desktop app, llama serve is already running. You can check by browsing the model list on localhost:9931.
If not, start llama serve. Passing a model starts single-model mode instead of router mode. See the llama.cpp server guide for router configuration.
You have two options:
- Automatic loading: by default, models load on demand.
- Manual loading: add
--no-models-autoload, then use/llamato load models before selecting them.
For automatic loading, just launch:
llama serve
In most cases you do not need any extra flags as the defaults should be reasonable. For advanced usage, here are a sample of possible flags you might be interested in:
-ngl alloffloads all layers to the GPU. Omit it to use automatic GPU offloading.-c 32768sets the context size in tokens. Use-c 0for the modelβs full context window, which may require substantially more memory. Context is shared across parallel server slots.--models-dir ~/modelsdiscovers local GGUF files from a custom local directory.
While the above command should be all that you need, but here are some common flags to use with llama serve:
| Flag | What it does |
|---|---|
-ngl all | offloads all layers to the GPU |
-c 32768 | sets the context size in tokens |
--models-dir ~/models | discovers local GGUF files from a custom local directory |
5. Connect Your Agent
Pick one of the agents below and follow the setup instructions.
Pi
Pi is the agent behind OpenClaw and is now integrated directly into Hugging Face, giving you access to thousands of compatible models.
Install Pi:
npm install -g --ignore-scripts @earendil-works/pi-coding-agent
Start Pi in your project directory:
pi
Configure llama.cpp:
/login llama.cpp
Run /llama to load a model, then /model to select the loaded model for the current session.
OpenClaw
OpenClaw works locally with llama.cpp. You can set your model via the onboard command:
openclaw plugins install @openclaw/llama-cpp-provider openclaw onboard
Local Memory Search for OpenClaw
You can run local embedding models with Llama.cpp for your agentβs memory search. To do so, make sure to have node-llama-cpp.
npm i node-llama-cpp
Hereβs an example snippet to run quantized EmbeddingGemma-300M locally for memory search. OpenClaw automatically downloads and serves the model with the command below.
openclaw config set agents.defaults.memorySearch.provider local
openclaw config set agents.defaults.memorySearch.local.modelPath "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf"Restart the gateway and validate.
openclaw gateway restart
openclaw memory status
# Memory Search (main)
# Provider: local (requested: local)
# Model: hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.ggufHermes Agent
Hermes Agent works locally with llama.cpp. In ~/.hermes/config.yaml add the following.
custom_providers:
- name: llama cpp (127.0.0.1:8080)
base_url: http://127.0.0.1:8080/v1Local Memory Search for Hermes Agent
Hermes Agent consumes semantic search models through endpoints. Once you get your preferred embedding model up on endpoint 8080 with llama.cpp or the inference engine of your choice, add the following to ~/.hermes/config.yaml.
auxiliary:
session_search:
base_url: "http://127.0.0.1:8080/v1"
api_key: "no-key-required"
model: "local-llama" # your model alias
timeout: 90
max_concurrency: 1Check if this works, none - built-in only shows that no other memory plug-ins are used. The output below shows that local serving is active.
$ hermes memory status
# Memory status
#ββββββββββββββββββββββββββββββββββββββββ
# Built-in: always active
# Provider: (none β built-in only)OpenCode
OpenCode works locally with llama.cpp. Define a ~/.config/opencode/opencode.json:
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"llama.cpp": {
"npm": "@ai-sdk/openai-compatible",
"name": "llama-server (local)",
"options": {
"baseURL": "http://127.0.0.1:8080/v1"
},
"models": {
"gemma-4-26b-4b-it": {
"name": "Gemma 4 (local)",
"limit": {
"context": 128000,
"output": 8192
}
}
}
}
}
}Alternative: llama-agent
llama-agent takes a different approach β it builds the agent loop directly into llama.cpp as a single binary with zero external dependencies. No Node.js, no Python, just compile and run:
git clone https://github.com/gary149/llama-agent.git
cd llama-agent
# Build
cmake -B build
cmake --build build --target llama-agent
# Run (downloads the model automatically)
./build/bin/llama-agent -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_MBecause tool calls happen in-process rather than over HTTP, there is no network overhead between the model and the agent. It also supports subagents, MCP servers, and an HTTP API server mode.
Next Steps
- Use AI Models Locally β Learn more about running models on your machine
- llama.cpp Guide β Detailed llama.cpp installation and usage
- Agents on the Hub β Connect agents to the Hugging Face ecosystem