Research March — April 2026
Distillation Reasoning Transfer
Can a small model learn to reason from a big one, using nothing but the big model's outputs? I built a black-box distillation framework to find out, small enough to train on consumer hardware, and ran the ablations to show what actually transfers.
Project report
The full write-up: method, ablations, and results. Open the PDF in a new tab, or read it below.
The question
Frontier models reason well but are expensive and remote. Small models are cheap and local but reason poorly. Distillation promises to move capability from one to the other, but with a black-box teacher you only get its text, not its weights or logits. How much reasoning survives that?
What I built
A teacher–student framework where the teacher is a frontier LLM behind an API and the student is a small open model fine-tuned locally. Fine-tuning uses QLoRA/PEFT with 4-bit quantization, so the whole loop runs on consumer hardware.
The teacher generates chain-of-thought traces; only traces that reach the correct answer are kept, so the student learns from reasoning that actually worked.
What it measured
Distilling 3,000 correctness-filtered chain-of-thought traces from Claude Haiku into Qwen2.5-1.5B raised GSM8K accuracy from 30% to 46%, closing about 46% of the gap to the teacher.
What actually transfers
A better score isn't an explanation, so I ran controlled ablations: train on answers only, on reasoning only, and on full chain-of-thought. The reasoning traces were the driver of the transfer, not the answers.
An LLM-as-judge pipeline scored 500 student outputs for quality and faithfulness, to check that the student's reasoning was sound and not just its final answers.
In short
- End-to-end research project: hypothesis, framework, training, evaluation, ablations.
- Comfortable with the practical side of fine-tuning: PEFT, quantization, memory budgets, consumer GPUs.
- Isolated the mechanism, not just the headline number.