According to different characters of infrared image and panchromatic image, a new image registration method was proposed based on the gradient structure information of the image. Firstly, the gradient of infrared image and panchromatic image were calculated. Then the structure similarity between two gradient images was measured to generate homologous points. RANSAC was used to robustly remove the wrong homologous points. Finally, after dividing image into small regions on triangle irregular network (TIN), the registered image was obtained by transforming the triangle. Experiment results show that the proposed approach is efficient and the register accuracy is high.