In the application of infrared super-resolution imaging, it is actually necessary to improve the subjective visual effects of images. At present, most image super-resolution reconstruction methods based on deep learning are trained and optimized with objective evaluation index as loss function. The subjective evaluation methods are difficult to apply due to the difficulty of quantification. Therefore, this paper focuses on the correlation between subjective evaluation and quantifiable objective evaluation indexes, and finds that the characteristics of phase consistency are highly correlated with subjective evaluation results. Based on this, a loss function based on subjective and objective joint evaluation is designed and applied to the super-resolution reconstruction algorithm of infrared image. Experiments show that this method can improve the subjective visual effect of image while maintaining the objective evaluation score.