Quickstart | Distil Labs

This guide will walk you through training your first specialized small language model (SLM) with distil labs in just a few steps

Prerequisites

A distil labs account
Python 3.7+
tar
Basic understanding of your task requirements

Authentication

First, set up authentication to access the distil labs API. You can get the API token using the snippet in Account and Authentication.

1 auth_header = {"Authorization": f"Bearer {token}"}

Data preparation

A training job requires three main components:

Configuration file: YAML defining model and training parameters
Job description: JSON file describing your task in plain English
Training and testing data: Small dataset (10-50 examples) showing inputs and expected outputs. They should be tables with question, answer columns as well as an optional context column.
Unstructured data: JSONL file containing unstructured text data to aid is synthetic training data generation

Step 1: Prepare and upload data

1 import json
2 import time
3 import requests
4 
5 DISTIL_API = "https://api.distillabs.ai"
6 auth_header = {"Authorization": f"Bearer {distil_bearer_token()}"}
7 
8 data = json.dumps({
9     "config.yaml": open("data/config.yaml").read(),
10     "job_description.json": open("data/job_description.json").read(),
11     "test.jsonl": open("data/test.jsonl").read(),
12     "train.jsonl": open("data/train.jsonl").read(),
13     "unstructured.jsonl": open("data/unstructured.jsonl").read()
14 })
15 response = requests.post(
16     f"{DISTIL_API}/uploads",
17     headers={"content-type": "application/json", **auth_header},
18     data=data,
19 )
20 
21 upload_id = response.json()["id"]
22 print(f"Upload successful. ID: {upload_id}")

Step 2: Teacher evaluation

Before training an SLM, distil labs validates whether a large language model can solve your task:

1 response = requests.post(
2     f"{DISTIL_API}/teacher-evaluations/{upload_id}",
3     headers=auth_header
4 )
5 eval_job_id = response.json()["id"]
6 print(f"Teacher evaluation (~5 minutes). ID: {eval_job_id}")
7 
8 running = True
9 while running:
10     response = requests.get(
11         f"{DISTIL_API}/teacher-evaluations/{eval_job_id}/status",
12         headers=auth_header
13     )
14     status = response.json()["status"]
15     if status != "JOB_RUNNING":
16         running = False
17     print(f"Evaluation status: {status}, re-checking in 10s...")
18     time.sleep(10)
19 
20 # Check evaluation results
21 print(f"Status: {response.json()}")

Step 3: SLM training

Once the teacher evaluation completes successfully, start the SLM training:

1 response = requests.post(
2     f"{DISTIL_API}/trainings/{upload_id}", headers=auth_header
3 )
4 training_job_id = response.json().get("id")
5 print(f"Distilling model (~20 minutes). ID: {training_job_id}")
6 
7 running = True
8 while running:
9     response = requests.get(
10         f"{DISTIL_API}/trainings/{training_job_id}/status",
11         headers=auth_header
12     )
13     status = response.json()["status"]
14     if status != "JOB_RUNNING":
15         running = False
16     print(f"Training status: {status}, re-checking in 30s...")
17     time.sleep(30)
18 
19 # When complete, check performance
20 response = requests.get(
21     f"{DISTIL_API}/trainings/{training_job_id}/evaluation-results",
22     headers=auth_header
23 )
24 print(f"Evaluation results: {response.json()}")

Step 4: Download your model

Once training is complete, download your model for deployment:

1 response = requests.get(
2     f"{DISTIL_API}/trainings/{training_job_id}/model",
3     headers=auth_header
4 )
5 print(f"Model ready for download at: {response.json()}")

Step 5: Deploy the model

After you download and untar the model, you can easily deploy it with any model-serving library of your choosing. The following command starts a vllm server with the fine-tuned model:

$ vllm serve model --api-key EMPTY

Query the model with input prompts:

1 from openai import OpenAI
2 
3 
4 client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
5 messages = [
6     {"role": "user", "content": "Hey, can you help me?"},
7     {"role": "system", "content": "You are a helpful assistant."},
8 ]
9 chat_response = client.chat.completions.create(
10     model="model",
11     messages=messages,
12 )
13 print(chat_response)

Next steps

That’s it! You have successfully trained and deployed a specialized small language model that performs your specific task with high accuracy while being much smaller than general-purpose LLMs. For more advanced usage, explore our comprehensive How to documentation.