It is proved that the radiometric accuracy of short wave infrared (SWIR) hyperspectral imager and its signaltonoise ratio could be affected by the background radiation. The quantified computing model of background radiation of dispersive SWIR hyperspectral imager is given. The relationship between the temperature and the radiometric accuracy is analyzed, which was demonstrated by experiments. Based on the analysis of the characteristics of background radiation, several methods of reducing the influence of background radiation were summarized, such as cold shield of small relative aperture, temperature controlling with high precision, cold optics, regular calibration and so on. The results in this paper is helpful to improve the quantification accuracy of hyperspectral imager.