KD Capacity Gap · Interactive Explorer · v2
Student Capacity Moderates Knowledge Distillation Effectiveness Yaşar (2026) · arXiv:2605.31191 (v2) · CIFAR-10 · 4 ResNet teacher-student pairs · 5 seeds · validation-based protocol
Four Main Findings (v2)
① Student capacity moderates KD — the only statistically significant gains (p<0.05, Welch) belong to R34 students under Feature-KD (+0.19 / +0.21 pp). Doubling teacher size at fixed teacher accuracy (R50 → R101) leaves the gain unchanged: the moderating variable sits on the student side.
② Feature-KD ≥ Logit-KD in all four pairs — and Feature-KD students land closer to the teacher's T=1 output distribution (KL 0.14–0.19 vs 0.21–0.28) despite never observing teacher logits.
③ Fidelity decouples from accuracy — teacher-student top-1 agreement is flat (95.3–95.5%) across all cells, echoing Stanton et al. (2021): distillation gains ≠ closer imitation.
④ Architecture dominates KD — the CIFAR stem correction is worth +5.5 to +7.2 pp, more than 25× the largest KD gain.
Correction from v1: the gradient-clipping bug blamed in v1 for Feature-KD's underperformance had no measurable effect on controlled re-run (p=0.69; unclipped projection norms ≤0.21 vs threshold 1.0). v1's larger gains are explained by test-set hyperparameter selection.
Interactive Configuration Explorer
Select a teacher, student, and distillation method. Studied pairs: R50→R18, R34→R18, R50→R34, R101→R34.
About This Demo
This Space accompanies version 2 of the paper:
Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10 Umut Onur Yaşar · arXiv:2605.31191
Evaluation Protocol (v2)
- CIFAR-10 train split once into 45k train / 5k val (fixed split seed); test set used only for final reporting
- Stage 1: full hyperparameter grid, selection by validation accuracy
- Stage 2: best configs + baselines re-run with 5 seeds; paper reports test acc of best-val checkpoints
- Fidelity metrics on every final run: teacher-student top-1 agreement, KL(p_T ‖ p_S) at T∈{1,4}, per-class accuracy
Setup
- Teachers (3 seeds, 200 ep): R101 95.37%, R50 95.36%, R34 95.30%
- Baselines (5 seeds): R18 94.86 ± 0.14, R34 95.04 ± 0.13
- Methods: Logit-KD (KL on temperature-scaled logits, α∈{0.3,0.5,0.7}, T∈{2,3,4}); Feature-KD (MSE + cosine on 4-layer features via 1×1 projections, α∈{0.3,0.5,0.7}, β=0.5)
- Hardware: NVIDIA A100
What changed from v1
v1 selected hyperparameters and checkpoints on the test set with 3 seeds and 3 pairs. v2 introduces the validation protocol, 5-seed finals, a fourth pair (R101→R34), fidelity metrics, a measured stem ablation — and retracts v1's bug attribution after a controlled re-run.