Aiming at the problems such as insufficient learning ability of model features and low quality of generated image caused by the current scarcity of visible-infrared image datasets, a single-sample unsupervised learning method to train infrared image generation model is proposed in this paper. First of all, when the dataset is difficult to obtain, only a pair of visible-infrared images are used as the data for model training, which reduces the difficulty of data acquisition and solves the problem of data scarcity. Secondly, in order to fully extract image features during the training of the model, the network structure is improved. Experimental data show that good results can be achieved in single-sample image generation by the proposed method. In the InfiRay OE dataset, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the proposed method reach 26.5588 dB and 0.8846, respectively. PSNR and SSIM of the Ohio State University (OSU) dataset reach 30.3528 dB and 0.9182, respectively. Compared with the style-based generative adversarial network (StyleGAN) method, PSNR and SSIM of the proposed method in the InfiRay OE dataset are increased by 16.07% and 23.78%, respectively. PSNR and SSIM of OSU dataset are increased by 31.8% and 40.4%, respectively. The results show that the image quality evaluation index of the proposed method is improved significantly, and the texture details of the generated infrared image are rich and close to the real infrared image. The research has a certain reference significance for the optimization of infrared image generation technology in the future.