In order to solving the problem of multi-target detection of infrared image in urban background by traditional methods, migration learning technology is used to migrate the target detection framework of visible light in deep learning to the infrared domain. A model is built by the method. The model''s small target detection performance is very good, and the average precision mAP(IoU=0.50)of the test set is 0.858 on the produced test set. A preliminary study of the relationship between training data and model detection performance was also conducted. Two training sets of large data volume and small data volume were produced, the model was trained, and then tested on the same test set. The average precision mAP(IoU=0.50)of the small data set is 0.615. The experimental results show that the diversity, quantity and quality of the data will affect the quality of the model.