With the wide application of infrared focal plane arrays (IRFPA), the requirements of detector assemblies in volume and fast cool-down etc are becoming higher and higher. To solve this problem, the influence factors on the fast cool-down of IRFPA detectors are analyzed on the basis of the design of detector chip, Dewar structure and fast cool-down cooler. The conclusions which are helpful to the improvement of fast cool-down of IRFPA detectors are drawn.