Does the code actually run?
The model writes a function; the extracted code is executed against tests in an isolated sandbox. Pass means exit 0 — no partial credit for looking right.
§LLM evaluation · measured 2026-07-16
How much model do you actually need?
Eleven language models — small local ones, a brand-new Czech reasoning model, and two
cloud references — against nineteen practical tasks in four disciplines.
Graded by code, not by vibes: temperature 0, fixed seed, no LLM judging LLMs.
01/ The four disciplines
Tasks a working assistant actually faces.
Each purpose is scored 0–1 by a deterministic grader. Code either runs or it doesn't; JSON either parses or it doesn't; a quote is either in the source or it isn't.
Coding
Triage
Can it make a clean call?
Given an operational digest, emit an escalate/don’t-escalate decision as machine-readable JSON. Scored against a labeled expectation.
Grounded
Does it stick to the source?
Summarize with claims backed by verbatim quotes. Score is the fraction of claims whose quote is actually present in the source text.
Structured
Will it obey a schema?
Emit JSON matching a given schema — required keys, types, constraints — with nothing else around it. The signal for automation-safety.
02/ Results
The matrix.
Scores are per-discipline averages over that discipline's tasks; overall is the mean across all nineteen.
| Model | coding | triage | grounded | structured | overall | med s |
|---|---|---|---|---|---|---|
| Qwen 2.5 7B | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1 |
| Qwen 2.5 14B | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1 |
| GPT-OSS 20B | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 2 |
| Qwen 2.5 Coder 7B | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1 |
| ThinkingCap 27B reasoning · BottleCap AI | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 16 |
| Claude Haiku cloud | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1 |
| Claude Sonnet cloud | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 2 |
| Qwen 3 8B reasoning | 1.00 | 1.00 | 0.94 | 1.00 | 0.99 | 4 |
| Qwen 3 Coder 30B | 1.00 | 0.80 | 1.00 | 1.00 | 0.95 | 1 |
| Mistral Nemo 12B | 0.83 | 0.80 | 0.89 | 1.00 | 0.87 | 1 |
| Llama 3.1 8B | 0.83 | 0.80 | 1.00 | 0.75 | 0.84 | 1 |
Measured 2026-07-16 · 19 tasks × 11 models, zero call errors · temperature 0, fixed seed, deterministic code graders · med s = median seconds per task; local models on consumer dual-GPU hardware, cloud models via their provider API — latencies are indicative, not comparable across serving conditions.
03 / Reading the numbers
What the matrix does and doesn't say.
Small local models saturate these tasks. A 7B model running on consumer hardware passes every practical test here — the same score as the cloud references. For this class of work the question isn't "which model is smartest" but "what's the cheapest thing that never drops the format".
Reasoning models get punished by naive parsing — almost unfairly. ThinkingCap 27B (BottleCap AI's Apache-2.0 fine-tune of Qwen 3.6 27B, the new Czech release) initially scored 0.47 on this suite. Every single failure turned out to be a harness artifact: the model thinks out loud before answering, and the graders were parsing the thinking instead of the answer. With visible reasoning separated before grading, it scores a clean 1.00 — at 16 s median latency, the price of thinking. The earlier Qwen 3 8B scores published by others likely suffer the same effect; here its median dropped from 36 s to 4 s and coding went perfect once reasoning was handled properly.
The failures that remain are real. Llama 3.1 8B keeps missing a required import and breaks schemas (0.75 structured); Mistral Nemo and Qwen 3 Coder 30B misjudge two triage calls each. Those persisted across both harness versions — that's model behaviour, not measurement noise.
Limitations, honestly. These tasks are now too easy to separate the top models — a perfect row means "suitable", not "best possible". Size classes differ (7B to 27B plus cloud), so this is a suitability screen, not a fairness contest. And latency depends on serving conditions; treat the medians as orders of magnitude. Negative results and corrections stay published — the 0.47 story above is the method working, not failing.