The mean shift algorithm is a nonparametric statistical method for seeking the nearest mode of a point sample distribution. In the color image sequence, the mean shift algorithm is an efficient method for tracking object. However, there is a singular grey space for representing the infrared object in the infrared object tracking scenario. Due to the lack of the information for the object representation, the object tracking based on the mean shift algorithm may be lost in the infrared sequence. To overcome this disadvantage, a new scheme that is to construct a cascade grey space is proposed. Moreover, for the different infrared image sequence, different strategies are used to generate different cascade grey spaces. The experimental results of two different infrared image sequences show our new scheme is efficient and robust for the infrared small object tracking and infrared object in the severe clutter background tracking.