SLMs & fine-tuning
SLM Lab and li-distill: Small-Model Fine-Tuning Labs
A fine-tuned 1.5B model scored 0.894, lost to stock Qwen3-8B, and was not shipped.
These two labs ask whether a small local model can take over high-volume chores more cheaply and privately than a frontier model. The first answer I measured was no: my fine-tuned model lost to a larger stock model, so I did not package it.
The problem
Many AI jobs are high-volume and repetitive: sentiment on reviews, ticket triage, attribute extraction, drafting posts in a fixed style. Paying a frontier model for each call is expensive, and it sends the data out of the building. A small model on my own machine is cheap and private. The open question is whether it is good enough.
I wanted a place to test that question the same way each time, with a fixed evaluation set and a clear bar to clear.
What I built
SLM Lab is a QLoRA fine-tuning workbench built on PyTorch, transformers, peft and bitsandbytes, with models served through Ollama. For one adapter I also used llama.cpp to serve LoRA adapters on the unchanged Qwen3-1.7B weights. The method has four guards. Training data is synthetic, and it is decontaminated against the evaluation set. The evaluation set is blind-audited. A regression gate decides whether a result is allowed to ship. Teachers were Qwen3-8B locally, gpt-oss-20b on a rented RunPod GPU, and Claude Sonnet for synthetic data.
The first task was review sentiment. The evaluation set is 160 gold items, and a blind audit agreed with the gold labels on all 160.
li-distill applies the same idea to writing. A larger teacher, gpt-oss-20b, writes example LinkedIn-style posts, and LoRA adapters teach Qwen3-1.7B to produce them.
How it performed
On review sentiment, stock Qwen3-8B scored 0.943. My fine-tuned Qwen2.5-1.5B, trained on 500 synthetic examples in 346 seconds using 3.81 GiB of VRAM, scored 0.894. It fell below the bar, so I did not package it. A run on 1,400 examples and a run on a 4B model are queued.
For li-distill, the general-marketing adapter trained on 861 examples. The management-posts adapter’s held-out loss fell from 3.04 to 0.90 in 23.6 minutes. Loss is a training measure rather than a quality judgment, so the claim here is that the adapter learned the target distribution, not that its output is good.
One result from this family did ship: the author-query triage model described on its own page.
What I learned
A fine-tune is not a win until it beats the alternative you would otherwise use. A stock 8B model needs no training, and it beat my tuned 1.5B model. The tuned model was smaller, but it fell below the bar, so I did not package it. I count that as the lab working.
Decontaminating synthetic data against the test set is not optional, because otherwise the score flatters the model.
Loss going down is not evidence of quality. For the writing adapters I have a number that improves and no judgment from a reader. Until I have one, the right claim is that the training worked, not that the posts are good.
The lab’s value is the discipline: a fixed bar, an audited test set and a willingness to publish the loss.