To overcome the problem that the point target in infrared sequence images is more difficult to be imaged and tracked, the Hough transform algorithm is optimized in computational complexity according to the features of the infrared sequence images. The three dimensional space of the Hough transform is reduced to two dimensional space and a Hough transform-based method for tracking the point target in infrared sequence images is proposed. The experimental result shows that since the method can track the targets whichever have partial defects and size variation stably. Its stability in tracking infrared targets is improved.