Small-Model Distillation — Part 6: Matching GPT-5.5 Medium With a 9B Open Model

TL;DR

I post-trained Qwen3.5-9B on one bounded agent workload, multi-turn SQL repair, and moved it from 78 to 115 solved tasks out of 220. The benchmark was frozen before I started and never changed. GPT-5.5 medium scores 115 on the same 220 tasks.

Model	Solved	Accuracy
Qwen3.5-9B base	78/220	35.5%
After Qwen3.5-35B-A3B trace SFT	96/220	43.6%
After DeepSeek V4 Pro trace SFT	107/220	48.6%
After correction Round 1	112/220	50.9%
After correction Round 2	115/220	52.3%
GPT-5.5 medium (reference)	115/220	52.3%
DeepSeek V4 Pro (the teacher)	120/220	54.5%

The whole thing cost about 50 euros. Training ran on a rented A40 at $0.44 an hour, and once you count setup, restarts and evaluation passes it was 15 to 20 hours on the meter, so under 10 dollars. The rest was about 40 euros of OpenRouter credit for the DeepSeek teacher calls. Dataset preparation and every evaluation ran on my own Mac.

So the teacher data cost roughly four times the GPU rental. That surprised me, and I think it is the number that matters if you are weighing this for your own workload. At this scale the compute is nearly free. The supervision is the expense, and further down I show that most of the supervision I paid for got thrown away before it ever reached the model.

The benchmark never moved, and that is the whole point

The task is multi-turn SQL repair. The model gets a user issue and some buggy SQL. It can inspect the schema, run queries against a real SQLite database, read what comes back, and submit a corrected query. It has eight turns. A task counts as solved only when the submitted SQL passes hidden tests the model never sees.

The data comes from birdsql/six-gym-sqlite, filtered to Query tasks across four SQLite databases. That gives 879 training tasks and a held-out set of 220: books 56, chinook 50, movie_3 55, netflix 59.

The split, prompt, tools, parser, database execution, stop rules and scorer were identical for every number in this post. So was the evaluation set. And nothing from it entered training at any stage.

That fixed substrate is the boring part people skip, and it is what makes everything after it mean anything. If the prompt, tools, or scorer change between runs, a higher score may describe a different system rather than a better model.

Teacher data got me to 107, then stopped paying

The first two stages were the obvious thing. I collected successful trajectories from stronger models and trained on them. Qwen3.5-35B-A3B first, which took the student from 78 to 96. Then DeepSeek V4 Pro, continuing from that same 96-point adapter rather than restarting from the base model, which took it to 107.

Those stages taught the model what good decisions look like inside the full agent loop, including how to read a tool result and when to stop and submit. They also suggest a student is not capped at the score of any one teacher, because it can combine what it already knew with evidence from several sources.

But the second teacher bought me 11 tasks where the first bought 18, and I had no reason to think a third would do better. The problem is that a successful teacher trajectory starts from the original task. It shows one clean route through a problem the student may not even find hard. It says nothing about the specific mess the student’s own policy walks into on turn four.

So I stopped collecting good SQL and started collecting corrections. I call the loop verified failure-state correction, and it is a variant of DAgger: run the current model, find the states where it actually breaks, ask a stronger model to continue from exactly those states, and keep only what the environment verifies.

Two details matter here. The student’s failed action was never treated as a target. It was only used to locate the point where better supervision was needed. And every round mixed in successful behavior from the current model and from earlier ones, because each stage was adding capability and destroying capability at the same time.

One correction, start to finish

Here is a real one, from the netflix database.

The user wants the items with the most view summaries, and the count. Their SQL groups movie and season ids together and then ends with ORDER BY COUNT(a.f) DESC LIMIT 1.

The student inspected the schema, spotted a genuine bug, and fixed it. Null ids were being counted as a group, so it added WHERE movie_id IS NOT NULL to one half of the union and WHERE season_id IS NOT NULL to the other. Then it submitted, still ending in LIMIT 1.

It caught a real bug and still submitted the wrong answer. LIMIT 1 returns one row, and the question asks for the items that tie.

The teacher picked up from the state right before that submission. It did not guess. It ran a query first, replacing LIMIT 1 with a subquery that computes the maximum count and returns every row matching it. The database came back with two rows: item 11189 with 21 views, and item 11220 with 21 views. The tie was real and now visible. Only then did the teacher submit, and the hidden tests passed.

One more thing about this example is worth saying, because it shows the method is not magic. The collector’s first attempt branched at the last state before failure, and that continuation did not verify. It went back one turn and tried again from an earlier state, and that one did. Both attempts stayed inside the original eight-turn budget.

This is the whole argument for correcting at the student’s own state. The model did not need another generic lesson about ties in SQL. It needed a verified answer at the exact decision where its own policy went wrong, in a situation its own policy created.

803 teacher answers, 94 the database accepted

Round 2 ran the current model across all 879 training tasks. It solved 435 of them, and those successes were kept as retention data. The failures produced 870 candidate states worth correcting, roughly two per failed task. DeepSeek was asked to continue from 803 of them. The other 67 were skipped because a sibling state in the same task had already produced a verified correction, or because of an infrastructure error.

Ninety-four of those corrections passed verification. That is 11.7 percent.

I keep coming back to that number. It is the reason the teacher API cost four times the GPU rental, and it is also the reason the method works at all. A fluent continuation from a strong model was not good enough on its own. The database had to run it and the hidden tests had to pass, and almost nine times in ten they did not. Anything that failed was discarded rather than softened or partially credited.

How the final model compares

The student moved from 78 to 96 after Qwen3.5-35B-A3B trace SFT, then to 107 after DeepSeek V4 Pro trace SFT. Round 1 of correction reached 112, and Round 2 reached 115. The comparison that matters is the endpoint: the final 9B student matches GPT-5.5 medium on this workload and remains five tasks behind DeepSeek V4 Pro.

I should be precise about what 115 means. Checkpoints were selected by score on the same 220-task evaluation, because this workload has no separate validation split. Round 2 saved five of them, and they scored 108, 115, 114, 111 and 112. That variation is normal checkpoint selection, not a result of its own. I report the best, so read 115 as a selected score rather than an untouched holdout estimate.

Both models solve 115. They are not the same 115.

An aggregate tie is a weak claim, so here is the paired comparison on the same tasks.

Outcome	Tasks
Both solve it	96
Only Qwen3.5-9B	19
Only GPT-5.5 medium	19
Neither	86

Both models solve the same 96 tasks and disagree on 38, split evenly between them. Together they would solve 134. So the tie is real as an aggregate and misleading as a description: these are different models that happen to land on the same total, and 86 tasks defeat both of them.

The same thing shows up per database, where the 9B model wins two of four.

Database	Base	Final 9B	GPT-5.5
books	20	29	32
chinook	17	26	28
movie_3	24	32	31
netflix	17	28	24

DeepSeek V4 Pro, the teacher, scores 120 on this benchmark. The student it trained ends up 5 tasks behind it, and solves 13 tasks the teacher gets wrong. (That 120 merges the main run with a retry pass, and 7 tasks still ended in an infrastructure error rather than a wrong answer, so 120 is a floor and the real gap may be wider.)

The same score, at a small fraction of the cost to run

I never paid GPT-5.5 per token, because it was reached through a subscription and not a metered API. So this part is an estimate, and here is how it is built. I counted the characters in every prompt and every completion across all 220 tasks, from the stored traces of both runs, and turned them into tokens at four characters each.

GPT-5.5 used about 973 thousand input tokens and 54 thousand output tokens over 556 turns. At the published price of $5 per million input and $30 per million output that is $6.50 for the run, so about 5.7 cents for each of the 115 tasks it solved. Read that as a floor. It ran at medium reasoning effort, and reasoning tokens are billed at the output rate but never show up in the traces I kept.

Qwen3.5-9B is chattier. It took 904 turns and about 2.0 million tokens in total, where GPT-5.5 took 556 turns and about 1.03 million. I just do not pay per token for any of them. A published vLLM benchmark on the A40, the same card I rented for training, measures 8B-class models at around 1,250 tokens a second in FP16 with 50 requests in flight. Call it 1,000 for a 9B. That is about half an hour of GPU time for all 220 tasks, and the A40 rents at $0.44 an hour, so roughly 25 cents for the run, or 0.2 cents for each task it solved.

Both figures are estimates and the gap between them is more than twenty times. That number depends on keeping the GPU busy with many tasks at once, which is what running a 220-task evaluation looks like and is not what a quiet production endpoint looks like. One request at a time the throughput is far lower and the cost per task climbs. An idle GPU costs the same per hour as a working one.

The useful version of this is the break-even. The roughly 50 euros of training pays for itself after about a thousand solved tasks, and because the GPT-5.5 figure ignores its reasoning tokens, a thousand is the pessimistic end.

Every stage that gained also destroyed something

Net scores hide this completely. Round 2 looks like plus three. What actually happened is that it solved nine tasks it used to fail and broke six it used to pass. Every single stage did some version of that, and retention reduced the damage without ever eliminating it.

End to end it is the same story. The final model solves 43 tasks the base model could not, and loses 6 that the base model could. I would not have known about those six from the scores.

What is still broken

The model got noticeably better at operating the agent loop. It submits more often, repeats itself less, and hits the turn limit less. Parse failures stayed near zero throughout.

But the remaining failures are semantic. 72 tasks reached a submission and produced SQL that failed the hidden tests. The next round of work would have to improve decisions about joins, grouping, ties and edge cases, and no amount of cleaner formatting will touch that.

GPT-5.5 medium fails 105 of the same 220 tasks, and every one of those is a submission that failed the tests. It never ran out of turns, never repeated itself, and never wrote anything the parser rejected. So all of its remaining errors are semantic, while 33 of mine are still about operating the loop. The score is the same and the work left to do is different.

What did not work

Before the correction rounds I tried reinforcement learning. Plain GRPO from the 96-point checkpoint scored 95 out of 220, so slightly below where it started. The reason was visible in the rollouts: most sampled groups contained no successful trajectory at all, which means no variation in reward inside the group and nothing for the algorithm to learn from. Success was concentrated in the tasks the model already found easy, so most groups came back uniformly failed and there was no gradient worth following. That checkpoint is not an ancestor of anything in this post.

I also ran two DeepSeek recovery passes at different reasoning effort settings to rescue states the first pass had failed on. Both produced zero verified traces out of 328 attempts.

What I would tell someone repeating this

Two things did most of the work.

The first is that the evaluation is the product. A model can get better at predicting training examples while getting worse at the actual job, and the only thing that caught that here was running all 220 tasks end to end. It is what measured the final model and exposed the six tasks quietly lost along the way.

The second is that verification matters far more than how confident the teacher looks. 803 teacher continuations went in and 94 came out. Had I trained on all of them because they looked reasonable, I would have taught the model seven wrong answers for every right one.

The SQL-specific parts of this were the task adapter and the verifier. The collection loop, the retention mixing, the audit and the checkpoint selection are the same code that would run on a different workload.

What this does and does not show

It shows that a compact open model can reach the same score as a much larger frontier model on one bounded workload, for about 50 euros, when the data, the correction loop and the evaluation are all built around that workload.

It does not show that Qwen3.5-9B is generally equal to GPT-5.5 medium. It is not. It shows that on this job, measured this way, the gap closed.

What comes next

The next useful proof is a real workload with a frozen baseline: whatever model you run today, the quality it reaches, what it costs per completed task, and how long it takes. Then the same loop answers a practical question. Can a compact model you own do that one job comparably, at economics that justify running it?

References

• Ross, Gordon, and Bagnell. “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.” AISTATS, 2011.
• Ai2. “Tülu 3: The next era in open post-training.” 2024.