Study · task distillation
A frontier LLM's judgment, in a model that runs on a laptop.
A 72-billion-parameter open LLM reads real, messy tweets and labels them. With no human annotation, mixle distills those judgments into a tiny model — a ~1,500-parameter head over a frozen 109M encoder — that keeps 85–91% of the frontier model's accuracy, classifies a tweet in ~3 ms on a CPU, and on the noisiest inputs is the more robust of the two.
What makes this real
Distillation demos usually run on clean academic benchmarks. This one is built to be checked: real user text, an honest frontier baseline, no gold labels leaking into the student, and the same corrupted inputs fed to teacher and student so the robustness comparison is fair.
Real messy data. tweet_eval — genuine tweets with typos, slang, emoji, @mentions, and hashtags across emotion, hate speech, and irony.
An honest frontier. Qwen2.5-72B-Instruct classifies zero-shot with constrained decoding, so every label is valid. Its accuracy on the gold test set is the bar.
No human labels. The student trains only on the 72B's predictions over an unlabeled pool. Gold labels are used for evaluation, never for training.
Fair noise. Realistic keyboard-typo corruption is applied once and the exact same strings go to both models — nobody sees a cleaner version.
Reported in full. Where the frontier wins, the charts show it. Macro-F1 (the tweet_eval metric) on held-out test, not cherry-picked accuracy.
Built from mixle. mixle.task.llm_labeler for the teacher, a mixle softmax head and distill_from_labels for the students, confidence gating for the cascade.
The match
How close does the tiny model get?
Macro-F1 on the clean gold test set. The student is a mixle softmax head over frozen bge-base (109M) embeddings; the char model is a mixle.task hashed character-n-gram classifier with no encoder at all. Both learn only from the 72B's labels.
| task | frontier | student | char | recovered | 72B vs gold |
|---|
Robustness
Then the text gets messier.
Macro-F1 as a realistic keyboard-typo rate rises from clean to 35% of characters. Frontier LLMs are not immune to jank — the 72B loses up to a quarter of its emotion F1. On hate speech and irony the tiny models degrade less than the 72B; the encoder-free character model is the most noise-stable of all.
Relative F1 lost to heavy noise
clean → 35% typosLower is more robust. On hate and irony the tiny models hold on better than the frontier; on emotion the 72B's scale keeps it steadier.
| task | frontier | student | char |
|---|
The cascade
Recover the rest for a few percent.
The student knows when it is unsure. Keep its answer when it is confident, and escalate only the least-confident inputs to the 72B. On emotion, escalating the hardest 32% lifts F1 from 0.66 to 0.74; escalating half puts it within two points of the frontier — while the tiny model still handles the majority itself.
This is mixle.task cascade economics: a calibrated confidence gate over the local
model, the frontier as fallback, and a cost model that tells you where to set it.
Emotion: escalate vs macro-F1
target 0.78Frontier accuracy at a fraction of the frontier calls.
Economics
Why the small model matters.
The teacher is a 72B model that needs a GPU and about 40 GB of memory to answer. The student is a frozen 109M encoder plus a linear head of about 1,500 numbers, and it answers on any laptop CPU. It reproduces the 72B's exact call 71–77% of the time.
| frontier 72B | student | char model | |
|---|---|---|---|
| parameters | 72,000M | 109M + 1.5k | ~1M |
| hardware | GPU, ~40 GB | laptop CPU | laptop CPU |
| per tweet | tokens on a GPU | 3.25 ms | 0.09 ms |
| throughput | — | ~300/s (1 core-set) | ~10,000/s |
| training labels | — | 0 human | 0 human |
How it is built
The whole pipeline is mixle.
from mixle.task import *
labels = ["anger", "joy", "sadness"]
# unlabeled_pool = ["great launch win", ...]
# requests = ["the fix is broken", "sad to miss it", ...]
teacher = llm_labeler(
OpenAICompatLLM(
"http://localhost:11434/v1",
"llama3.1",
),
labels,
)
teacher_labels = teacher(unlabeled_pool)
cascade = Cascade(
CalibratedTaskModel(
distill_from_labels(
unlabeled_pool,
teacher_labels,
labels=labels,
),
).calibrate(unlabeled_pool[:60], teacher_labels[:60]),
teacher,
cost=CostModel(c_frontier=0.01),
)
cascade.serve(requests)
cascade.report()Where the frontier still wins
The honest edges.
- Emotion is a scale task. On four-way emotion the 72B leads by 12 points clean and degrades more slowly — nuance across four classes is where the extra capacity pays off. The student recovers 85%, and the cascade closes the rest.
- Irony is hard for everyone. The 72B agrees with the gold labels only 67% of the time on irony; sarcasm resists zero-shot scale. The student nearly matches it precisely because the ceiling is low.
- The char model trades accuracy for robustness. With no encoder it starts several points lower, but it is the most stable under noise and runs at 10,000 tweets a second — the right pick when inputs are filthy and budgets are tiny.
- Distillation inherits the teacher. The student can only be as right as the 72B's labels; its ceiling is the frontier's judgment, not ground truth. The cascade is how you buy back the difference.