Remote sensing image scene classification aims to automatically assign a semantic label to each remote sensing image according to its content, and has become one of the hot topics in the field of remote sensing image processing. Methods based on convolutional neural networks (CNNs) and methods based on self-attention mechanism are two mainstream methods in remote sensing image scene classification. However, the former is less effective in exploring long-range contextual information, and the latter has limitations in learning local information and has a large number of parameters and calculations. In order to address these issues, a lightweight method based on knowledge distillation is proposed to solve the problem of scene classification for remote sensing images. The proposed method uses Swin Transformer and lightweight CNNs as the teacher model and the student models, respectively, and integrates the advantages of the two kinds of models by means of knowledge distillation. Furthermore, a novel distillation loss function is proposed to enable the student models to focus on both inter- and intra-class potential information of remote sensing images simultaneously. The experimental results on two large-scale remote sensing image datasets demonstrate that the proposed method not only achieves high classification accuracy compared to existing methods but also has a significantly reduced number of parameters and calculations.