To meet the requirements of high sensitivity, real time, miniaturization and integration of an infrared camera, an image data processing system based on FPGA is designed. The system uses the four-channel output mode of a 384×288 long-wave infrared focal plane array detector, takes a chip in Altera cyclones family as the main controller and uses a 4 M bit SRAM and a 4 M bit flash as the auxiliaries. It can implement the collection and elimination of image background, oversample-superposition and digital TDI etc through instruction. The system also has the functions of push-broom scanning and staring imaging which are achieved by Verilog language. Its transplantation is strong. The experimental results show that the system can implement real-time processing of images instead of PC and obtain clear images with high sensitivity.