A real-time target detection method based on center points is proposed for infrared imaging systems equipped with CPUs. Following the lightweight design principles， a backbone with low computational cost is first introduced for feature extraction. Correspondingly， an efficient feature fusion module is designed to exploit spatial and contextual information extracted from multi-stages. In addition， an auxiliary background suppression module is proposed to predict foreground regions to enhance the feature representation. Finally， a simple detection head predicts the target center point and its associated properties. Evaluations on the infrared aerial target dataset show that our proposed method achieves 90.24% mAP at a speed of 21.69 ms per frame on the CPU. It surpasses the state-of-the-art Tiny-YOLOv3 by 10.16% mAP with only 21% FLOPs and 11% parameters while also runs 10.02 ms faster. The results demonstrate its great potential for real-time infrared applications.