text-embeddings-inference documentation
Using TEI locally with CPU
Using TEI locally with CPU
You can install text-embeddings-inference locally to run it on your own machine. Here are the step-by-step instructions for installation:
Step 0: Clone the repository
We need the source code of the text-embeddings-inference repo to install it later. Clone it with:
git clone https://github.com/huggingface/text-embeddings-inference.git cd text-embeddings-inference
Step 1: Install Rust
Install Rust on your machine by run the following in your terminal, then following the instructions:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Step 2: Install necessary packages
The following assumes you are in the repository folder you cloned in step 0. Depending on your machine’s architecture, run one of the following commands:
For x86 Machines
cargo install --path router -F mkl
For M1 or M2 Machines
cargo install --path router -F metal
Source builds use native CPU optimizations by default. Build on the machine that will run TEI,
or on a CPU with compatible instruction sets. To disable the repository’s native CPU flags
when building for other CPUs of the same target architecture, prefix the install command with RUSTFLAGS=''. For example, with the ONNX CPU backend.
RUSTFLAGS='' cargo install --path router -F ort
This does not cross-compile the application or change the requirements of prebuilt backend libraries.
Step 3: Launch Text Embeddings Inference
Once the installation is successfully complete, you can launch Text Embeddings Inference on CPU with the following command:
model=Qwen/Qwen3-Embedding-0.6B text-embeddings-router --model-id $model --port 8080
In some cases, you might also need the OpenSSL libraries and gcc installed. On Linux machines, run the following command:
sudo apt-get install libssl-dev gcc -y
Now you are ready to use text-embeddings-inference locally on your machine.
If you want to run TEI locally with a GPU, check out the Using TEI locally with GPU page.