According to the particularity of infrared and low light image registration, to reduce the amount of registration, an Speeded Up Robust Feature (SURF) registration algorithm improved both in determination of main direction and in description of feature points is proposed. Firstly, the edges of low light images and infrared images are detected respectively. Then, the improved SURF algorithm is used to extract the feature points of two kinds of image edges. Secondly, a nearest neighbor method is used to screen out the original feature points. After the feature points with higher accuracy are obtained, rough matching is carried out on them. Then, the RANdom Sample Consensus (RANSAC) algorithm is used to carry out precise matching on the feature points screened out one time. Finally, the precise feature points are used to establish a transform model and the images to be registered after resampling are registered with the reference images. The experiment results show that this improved algorithm not only can solve the registration problem of infrared and low light images, but also has better performance than the original SURF algorithm.