Vibe Coder · Unit 19: Fine-Tuning a Model · Lesson 4 of 4
Is it better?
A tuned model has to beat what you already had. Compare it with the base model and with your best prompt on the hold-out set, count the cost, and publish the experiment.
Read the short notes and pictures.Do everything inside a blue frame.
The honest test
It is easy to be pleased with a model you trained yourself. The work only paid off if it beats what you already had. Often it does not, and finding that out cheaply is a good result: you keep the prompt, and you know why.
Three contenders

A: the base model
The small open model you started from, with a plain prompt. Shows what training added.
B: your best prompt
A strong hosted model with the careful prompt and examples from the ladder. This is what you would ship without fine-tuning.
C: the tuned model
Your small model with its adapter and the short system message it was trained with.
Judging
Run every hold-out input through all three
Save the outputs in a table.
Shuffle and hide the labels
So you do not know which model wrote which.
Score against a rubric
A few yes-or-no questions per output.
Reveal and total
Then read the failures of each contender, not only the totals.
A weak rubric
Is it good? (1 to 10)
A usable rubric
Correct facts from the input? yes / no Sounds like our voice? yes / no Right format and length? yes / no Invented anything? yes / no
Cost

Per call
A small tuned model with a short prompt uses far fewer tokens than a large model with a long one.
Speed
Small models answer faster, if the hardware is adequate.
Hosting
A hosted model is someone else’s problem. A tuned open model has to run somewhere: your own machine, a rented server or a provider that hosts open models.
Upkeep
When your style or policy changes, a prompt is edited in a minute. A tuned model is retrained.
Deciding
Ship the tuned model when it clearly wins on the hold-out set, or matches the prompt version at meaningfully lower cost at your volume, and when someone will own retraining it. In every other case keep the prompt. For most early products the answer is "not yet", and the dataset you built is still valuable: it is now your test set.
Results on 20 hold-out inputs: base model 8 pass, best prompt 17 pass, tuned model 16 pass. The tuned model costs a tenth as much per call. You handle 200 requests a day. What do you ship?
Compare, decide and publish
Run your hold-out inputs through the base model, your best prompt and your tuned model. Score them blind against a rubric, add the costs, and write your decision. Publish the experiment to GitHub.
Done when
Model files are large. Leave them out of the repository and say in the README where they are kept.
Keep these
- Compare the base model, your best prompt and the tuned model.
- Judge blind, with a yes-or-no rubric, on held-out inputs.
- Ship the tuned model only if it wins on quality or cost.