live cassette · 28 real eval runs · no server, no API calls
A fine-tuned 0.5B model outsprints Claude on its one job.
sql-specialist is a Qwen2.5-Coder-0.5B model, LoRA fine-tuned on 111 examples for one narrow task —
natural language to SQL over a 7-table e-commerce schema — then quantized to q4_k_m and served locally via Ollama.
Click a question below to replay its actual recorded run against Claude Haiku 4.5 prompted directly on the same task.
sql-specialist (local, fine-tuned)
Claude Haiku 4.5 (frontier, prompted)
metric
sql-specialist
claude haiku 4.5
Read the accuracy row next to the failure audit below before drawing conclusions from it — the gap is real but partly an eval-scoring artifact, not purely a reasoning gap. Latency and cost are not.
01 · Run a query
All 28 hold-out questions, with the specialist's real generated SQL, real latency, and real result rows from executing it against the live database.
02 · Accuracy by query shape
Where the frontier model actually falls down: multi-table joins, HAVING clauses, and scalar subqueries — the shapes the specialist saw most in its 111 training examples.
03 · The honest failure audit
Every one of Claude Haiku 4.5's 13 measured failures, read by hand — not just re-scored.
Zero were SQL logic errors. Every frontier failure was a column-selection or row-order
convention mismatch — extra columns "for context," a different sort order, an arguably-optional column
omitted. It never picked the wrong table, joined incorrectly, or miscounted. The specialist's two failures,
by contrast, were both genuine: a hallucinated orders.total column and a dropped table qualifier
— both shown in the console above (eval-0007, eval-0017). The accuracy gap is
real, but it's substantially the specialist having memorized this project's output conventions from its
training set, not raw text-to-SQL reasoning capability.
Qwen2.5-Coder-0.5B-Instruct
→
LoRA fine-tune, 111 examples
→
merge + GGUF q4_k_m
→
Ollama (local serving)
→
MCP tool: nl_to_sql