Ruixuan Zhang

Author directory

Other people with similar names: Ruixuan Zhang


2026

Model calibration aligns a model’s confidence with the accuracy of its predictions, directly influencing reliability and user trust. Surprisingly, existing research on knowledge distillation has paid little attention to the calibration of student models. In this paper, we evaluate calibration across diverse knowledge distillation settings and demonstrate that student models distilled from black-box teachers generally exhibit poor calibration. We attribute this to the fact that distillation fails to convey well-calibrated confidence, which serves as a crucial expression of predictive uncertainty. To address this limitation, we propose an ***U**ncertainty-**I**ntegrated **K**nowledge **D**istillation* (**UIKD**) method comprising two key components: (1) *uncertainty estimation*, which leverages the collective intelligence of multiple LLMs via a debate mechanism to quantify uncertainty in the distillation data; and (2) *uncertainty integration*, which employs an adaptive weighted loss function to integrate this data uncertainty into the student model effectively. Experiments on common-sense reasoning and natural language inference benchmarks demonstrate that UIKD consistently improves student model calibration while maintaining predictive accuracy. Our code is available at https://github.com/RuixuanZhang127/UIKD.