Visual Contrastive Self-Distillation
By Yijun Liang · Paper · cs.CV
On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student.