To detect dim and small infrared targets in real time, a real-time infrared target detection system based on DSP and FPGA high speed ping-pang cache structure is proposed. Taking a high speed cache structure as the core, the hardware system uses a region growing algorithm based on target characteristics to detect dim and small infrared targets. The experimental results show that the system has good detection effectiveness and high real-time performance for dim and small infrared targets. It can implement the real-time detection of dim and small infrared targets.