Model deployment

Once your model is trained, you can download and deploy it locally using the inference framework of your choice. Alternatively, you can push it to Hugging Face Hub for easy sharing and deployment.

Deploying with distil CLI

The distil CLI provides a built-in deployment command that handles model download and serving for you.

Local deployment (experimental)

Deploy your trained model locally using llama-cpp as the inference backend:

1
distil model deploy local <model-id>

This downloads the model and starts a local llama-server on port 8000. You can customize the port and enable server logs:

1
distil model deploy local --port 9000 --logs <model-id>

Once running, query the model using the OpenAI-compatible API at http://localhost:<port>/v1.

To get the command to invoke your locally deployed model:

1
distil model invoke <model-id>

This outputs a ready-to-run command using uv. Copy and run it directly:

$
uv run $PATH_TO_CLIENT --question "Your question here"

For question answering models that require context, use the --context flag:

$
uv run $PATH_TO_CLIENT --question "Your question here" --context "Your context here"

Downloading your model

Download your trained model using the CLI or API:

1
distil model download <model-id>

After downloading, extract the tarball. You will have a directory containing your trained SLM with all necessary files for deployment:

├── model/
├── model-adapters/
├── Modelfile
├── model_client.py
├── README.md

Deploying with vLLM

vLLM is a high-performance inference engine for LLMs. To get started, set up a virtual environment and install dependencies:

1
python -m venv serve
2
source serve/bin/activate
3
pip install vllm openai

Start the vLLM server:

1
vllm serve model --api-key EMPTY

For tool calling models:

1
vllm serve model --enable-auto-tool-choice --tool-call-parser hermes --api-key EMPTY

Note that model refers to the directory containing your model weights. The server runs in the foreground, so make sure to run the server in a separate window or a background process.

Query the model using the OpenAI-compatible API:

1
from openai import OpenAI
2
3
client = OpenAI(
4
    base_url="http://localhost:8000/v1",
5
    api_key="EMPTY",
6
)
7
8
response = client.chat.completions.create(
9
    model="model",
10
    messages=[
11
        {"role": "system", "content": "You are a helpful assistant."},
12
        {"role": "user", "content": "Your question here"},
13
    ],
14
)
15
print(response.choices[0].message.content)

Or use the provided client script:

1
python model_client.py --question "Your question here"

For question answering models that require context, use the --context flag:

$
python model_client.py --question "Your question here" --context "Your context here"

Pushing to Hugging Face Hub (API only)

You can upload your model directly to your private Hugging Face repository. Once pushed, you can deploy it directly from Hugging Face using various inference frameworks.

Requirements:

• Training ID of the model (YOUR_TRAINING_ID)
• Hugging Face user access token with write privileges (YOUR_HF_TOKEN)
• Repository name for your model (YOUR_USERNAME/MODEL_NAME)

1
import json
2
import requests
3
4
# See Account and Authentication for distil_bearer_token() implementation
5
auth_header = {"Authorization": f"Bearer {distil_bearer_token()}"}
6
7
slm_training_job_id = "YOUR_TRAINING_ID"
8
hf_details = {
9
    "hf_token": "YOUR_HF_TOKEN",
10
    "repo_id": "YOUR_USERNAME/MODEL_NAME"
11
}
12
13
response = requests.post(
14
    f"https://api.distillabs.ai/trainings/{slm_training_job_id}/huggingface_models",
15
    data=json.dumps(hf_details),
16
    headers={"Content-Type": "application/json", **auth_header},
17
)
18
print(response.json())

We push models to two repositories on Hugging Face, one in GGUF format and one in safetensors format. Once your model is on Hugging Face, you can deploy it directly using vLLM:

1
pip install vllm
2
vllm serve "YOUR_USERNAME/MODEL_NAME"

Production considerations

When deploying your model to production, consider:

• Resource Requirements: Even small models benefit from GPU acceleration for high-throughput applications.
• Security: Apply appropriate access controls, especially if your model handles sensitive information.
• Container Deployment: Consider packaging your model with Docker for consistent deployment across environments.