Author-Query Triage: A Fine-Tuned 1.7B Model

The problem

A scientific publishing services firm manages books largely by email and memory. Every chapter passes through authors, editors and production staff, and a steady stream of author queries arrives that someone must read and route. The work is repetitive, but the content is unpublished manuscript material, so sending it to a hosted model was not an easy answer.

My design for the wider engagement made the chapter the unit of work, so schedules, chasing and reports could be computed instead of remembered. Inside that design sat one narrow, high-volume chore: classify each author query. The question I wanted to answer was specific. Can a very small model, running on the firm’s own hardware, do this job well enough to matter?

What I built

The classification model is a Qwen3-1.7B base fine-tuned with QLoRA. Only 17.4 million of its 1.03 billion parameters were trained, about 1.69%. I exported the result to GGUF and served it through Ollama, so a call stays on the local machine.

The training labels were written by Claude, which makes this a distillation: a frontier model teaches a small one a single task. The 473 training examples were synthesised for the demonstration, not drawn from the firm’s live traffic. I say that plainly because it limits what the result proves.

Around the model sits a small Node.js demonstration application with three tiers. Rules handle what rules can handle. The local model handles classification. A frontier model is reserved for the work that genuinely needs it. The application also holds the schedule engine behind the chapter-as-unit idea. It rebuilds 186 tasks automatically, and in the demonstration one chapter running 20 days late moves publication from 30 December to 15 January.

I also benchmarked other small models against the same task, among them Qwen2.5-1.5B, Gemma 3 1B, Llama 3.2 1B and Phi-4-mini.

How it performed

I held out 94 queries that the model never saw in training, and scored three conditions on them:

• Base model, zero-shot: 36.2%
• Base model with a good few-shot prompt: 63.8%
• Fine-tuned model: 88.3%, which is 83 of 94 correct

The comparison that matters is against the strong prompt, not the zero-shot base. Fine-tuning added 24.5 points over a prompt I had worked to make good. Training took 19.3 minutes on an 8 GB laptop GPU with a peak of 3.86 GB. Local inference runs at roughly 150 to 200 milliseconds per call.

The limits are real. Ninety-four test queries is a small sample, so the interval around 88.3% is wide. The training data was synthetic. A third training epoch over-fitted. Hour savings for the client are estimates, because there was no baseline measurement of how long staff spent on this work. No deployment outcome is recorded: this was a working demonstration delivered as part of the engagement, not a system in production.

What I learned

The first lesson is about how to ask the question. A fine-tuned model compared with an unprompted base model always looks impressive, and tells you little. The honest comparison is with the best prompt you can write, and there the gain was still large. For a narrow, repetitive classification task with a clear label set, that gap is the argument for fine-tuning.

The second lesson is about where the effort goes. The training run took under twenty minutes, so the expensive parts of a project like this are the labels and the evaluation set, not the compute. Synthetic training data let me demonstrate the method quickly, but it is also the first thing I would replace with the firm’s real, reviewed queries before trusting the number in production.

The third is about the over-fit. More training was worse, which is a reminder to hold out data and watch it, rather than assume more epochs help.

Finally, a small model is not a replacement for a frontier model. It is the right tool for one well-defined chore, with a frontier model used in the design and the labelling, and rules handling what they can. The routing between the three tiers is as much the product as the model.