Knowledge distillation is a powerful technique for transferring knowledge
from a pre-trained teacher model to a student model. However, the true
potential of knowledge transfer has not been fully explored. Existing
approaches primarily focus on distilling individual information or
instance-level relationships, overlooking the valuable information embedded in
token-level relationships, which may be particularly affected by the long-tail
effects. To address the above limitations, we propose a novel method called
Knowledge Distillation with Token-level Relationship Graph (TRG) that leverages
the token-wise relational knowledge to enhance the performance of knowledge
distillation. By employing TRG, the student model can effectively emulate
higher-level semantic information from the teacher model, resulting in improved
distillation results. To further enhance the learning process, we introduce a
token-wise contextual loss called contextual loss, which encourages the student
model to capture the inner-instance semantic contextual of the teacher model.
We conduct experiments to evaluate the effectiveness of the proposed method
against several state-of-the-art approaches. Empirical results demonstrate the
superiority of TRG across various visual classification tasks, including those
involving imbalanced data. Our method consistently outperforms the existing
baselines, establishing a new state-of-the-art performance in the field of
knowledge distillation.