As a new image acquisition technology, the thermal infrared imaging can avoid the influence of illumination and background color on its image acquisition. Currently, the algorithms which use a thermal infrared sequence for target tracking mostly combine visible light images with thermal infrared images. In view of this limitation, a tracking algorithm based solely on the thermal infrared sequence is proposed. First, the background information and the moving direction of a target in the sequence are modeled by using a modified codebook algorithm. Then, the modeling result is incorporated into the compression tracking algorithm. The information on background, target motion and target characteristics is complemented one another in the tracking process. The experiment shows that the improved algorithm is fast in processing. Because its accuracy is improved greatly, it is adapted to the characteristics of thermal infrared sequences better.