Model training

After validating your teacher model’s performance, the next step is to train your small language model (SLM) using distil labs’ knowledge distillation approach.

Understanding knowledge distillation

Knowledge distillation is the core technology behind distil labs’ ability to create high-performing small models with minimal training data. The process works as follows:

1. Synthetic Data Generation: The large “teacher” model generates synthetic training data based on your problem definition, task description, and provided examples.
2. Synthetic Data Validation: We validate generated data to make sure the synthetic set is diverse and high-quality
3. Knowledge Transfer: The synthetic data is used to train the smaller “student” model with a loss function aligned with your specific task. This process enables the student model to emulate the teacher’s capabilities while maintaining a much smaller size.

Initiating model training

After completing teacher evaluation and confirming satisfactory performance, start the training process:

1
distil model run-training <model-id>

Monitoring training status

The training process typically takes several hours to complete. Check the current status of your training job:

1
distil model training <model-id>

Possible status values include:

• JOB_PENDING - Job is waiting to start
• JOB_RUNNING - Job is currently running
• JOB_SUCCEEDED - Job has finished successfully
• JOB_FAILED - Job encountered an error

Retrieving evaluation results

When the training is complete, retrieve detailed evaluation results to compare the performance of your trained SLM against the teacher model. See Metrics for details on each metric and how to interpret them.

1
distil model training <model-id>

What makes a successful training?

• Comparison to Teacher: Your SLM should achieve performance reasonably close to the teacher model (typically within one standard deviation)
• Task Requirements: The absolute performance should meet your specific application needs

If your SLM performance is significantly below the teacher model, consider:

1. Increasing the number of training examples
2. Adjusting your task description to be more specific
3. Modifying your configuration parameters (like increasing training epochs)
4. Using a slightly larger student model

Retrieving predictions (API only)

For more in-depth analysis, you can download the predictions on individual data points of the test dataset using the API. These predictions are generated using the fine-tuned student model.

1
print(response.json()["finetuned_student_evaluation_predictions_download_url"])

Download and read the predictions file:

1
curl -o finetuned_student_evaluation_predictions.jsonl "<DOWNLOAD_URL>"

The file is in JSON Lines format and can be read using:

1
import pandas as pd
2
df = pd.read_json("finetuned_student_evaluation_predictions.jsonl", lines=True)