Research March – April 2026
Distillation Reasoning Transfer
Can a small model learn to reason from a big one using only the big model's outputs? I built a black-box distillation setup that trains on a consumer GPU, then ran ablations to see what actually carries over.
Project report
The full write-up: method, ablations, and results. Open the PDF in a new tab, or read it below.
The question
Frontier models reason well but are expensive and remote. Small models are cheap and local but reason poorly. Distillation promises to move capability from one to the other, but with a black-box teacher you only get its text, not its weights or logits. How much reasoning survives that?
What I built
A teacher–student framework where the teacher is a frontier LLM behind an API and the student is a small open model fine-tuned locally. Fine-tuning uses QLoRA/PEFT with 4-bit quantization, so the whole loop runs on consumer hardware.
The teacher generates chain-of-thought traces; only traces that reach the correct answer are kept, so the student learns from reasoning that actually worked.
Results
Distilling 3,000 correctness-filtered chain-of-thought traces from Claude Haiku into Qwen2.5-1.5B raised GSM8K accuracy from 30% to 46%, closing about 46% of the gap to the teacher.
What actually transfers
A higher score doesn't explain itself, so I ran ablations: training on answers only, on reasoning only, and on full chain-of-thought. The gains came from the reasoning traces rather than the answers.
I also had an LLM judge score 500 student outputs for quality and faithfulness, to check the reasoning itself and not only the final answers.
What’s next
I’m following this up with LoopedReasoning, which tests whether two-pass draft-and-revise traces teach reasoning better than single-pass chain-of-thought.