Quickstart | Distil Labs

This guide will walk you through training your first specialized small language model (SLM) with distil labs in just a few steps

Prerequisites

A distil labs account
Python 3.7+
tar
Basic understanding of your task requirements

Authentication

First, set up authentication to access the distil labs API. You can get the API token using the snippet in Account and Authentication.

1 auth_header = {"Authorization": f"Bearer {distil_bearer_token()}"}

Data preparation

A training job requires three main components:

Configuration file: YAML defining model and training parameters
Job description: JSON file describing your task in plain English
Training and testing data: Small dataset (10-50 examples) showing inputs and expected outputs. They should be tables with question, answer columns as well as an optional context column.
Unstructured data: JSONL file containing unstructured text data to aid in synthetic training data generation

Step 1: Register a model

The first component of the workflow is registering a new model - this helps us keep track of all our experiments down the line.

1 import json
2 import requests
3 from pprint import pprint
4 
5 # Register a new model
6 auth_header = {"Authorization": f"Bearer {distil_bearer_token()}"}
7 response = requests.post(
8     "https://api.distillabs.ai/models",
9     data=json.dumps({"name": "testmodel-123"}),
10     headers={"Content-Type": "application/json", **auth_header},
11 )
12 pprint(response.json())
13 model_id = response.json()["id"]
14 print(f"Registered a model with ID={model_id}")

You can inspect your models with

1 from pprint import pprint
2 
3 response = requests.get(
4     "https://api.distillabs.ai/models",
5     headers=auth_header,
6 )
7 pprint(response.json())

Step 2: Prepare and upload data

1 import requests
2 
3 data = {
4     "job_description": {"type": "json", "content": open("data/job_description.json").read()},
5     "train_data": {"type": "csv", "content": open("data/train.csv").read()},
6     "test_data": {"type": "csv", "content": open("data/test.csv").read()},
7     "unstructured_data": {"type": "csv", "content": open("data/unstructured.csv").read()},
8     "config": {"type": "yaml", "content": open("data/config.yaml").read()},
9 }
10 response = requests.post(
11     f"https://api.distillabs.ai/models/{model_id}/uploads",
12     data=json.dumps(data),
13     headers={"Content-Type": "application/json", **auth_header},
14 )
15 upload_id = response.json()["id"]
16 print(f"Upload successful. ID: {upload_id}")

Step 3: Teacher evaluation

Before training an SLM, distil labs validates whether a large language model can solve your task:

1 import requests
2 import time
3 
4 data = {"upload_id": upload_id}
5 response = requests.post(
6     f"https://api.distillabs.ai/models/{model_id}/teacher-evaluations",
7     data=json.dumps(data),
8     headers={"Content-Type": "application/json", **auth_header},
9 )
10 
11 pprint(response.json())
12 eval_job_id = response.json()["id"]
13 print(f"Started teacher evaluation with ID: {eval_job_id}")
14 
15 running = True
16 while running:
17     response = requests.get(
18         f"https://api.distillabs.ai/teacher-evaluations/{eval_job_id}/status",
19         headers=auth_header
20     )
21     status = response.json()["status"]
22     if status != "JOB_RUNNING":
23         running = False
24     print(f"Evaluation status: {status}, re-checking in 10s...")
25     time.sleep(10)
26 
27 # Check evaluation results
28 print(f"Status: {response.json()}")

Step 4: SLM training

Once the teacher evaluation completes successfully, start the SLM training:

1 import requests
2 
3 data = {"upload_id": upload_id}
4 response = requests.post(
5     f"https://api.distillabs.ai/models/{model_id}/training",
6     data=json.dumps(data),
7     headers={"content-type": "application/json", **auth_header},
8 )
9 slm_training_job_id = response.json().get("id")
10 print(f"Training model with ID: {slm_training_job_id}")

You can check the status of your training with

1 import requests
2 from pprint import pprint
3 
4 response = requests.get(
5         f"https://api.distillabs.ai/trainings/{slm_training_job_id}/status",
6         headers=auth_header
7     )
8 pprint(response.json())

When the training completes, check performance with

1 import requests
2 response = requests.get(
3     f"https://api.distillabs.ai/trainings/{slm_training_job_id}/evaluation-results",
4     headers=auth_header
5 )
6 print(f"Evaluation results: {response.json()}")

Step 4: Download your model

Once training is complete, download your model for deployment:

1 response = requests.get(
2     f"https://api.distillabs.ai/trainings/{slm_training_job_id}/model",
3     headers=auth_header
4 )
5 print(f"Model ready for download at: {response.json()}")

Step 5: Deploy the model

After you download and untar the model, you can easily deploy it with any model-serving library of your choosing. You can find more information in model-deployment but for now, the following command starts an ollama server with the fine-tuned model:

$ ! ollama create model-distillabs -f model/Modelfile

Query the model with input prompts:

1 from openai import OpenAI
2 
3 
4 client = OpenAI(
5     base_url = 'http://localhost:11434/v1',
6     api_key='ollama', # required, but unused
7 )
8 messages = [
9     {"role": "user", "content": "Hey, can you help me?"},
10     {"role": "system", "content": "You are a helpful assistant."},
11 ]
12 chat_response = client.chat.completions.create(
13     model="model",
14     messages=messages,
15 )
16 print(chat_response)

Next steps

That’s it! You have successfully trained and deployed a specialized small language model that performs your specific task with high accuracy while being much smaller than general-purpose LLMs. For more advanced usage, explore our comprehensive How to documentation.