How to evaluate the reliabilities of different image sensors and their processing results is an important issue in the field of multi-modal fusion. In this paper, we focus on multi-modal fusion moving objects detection, in which visible light and infrared image sensors are adopted. An evaluation factor named F-BDEF (Foreground-Background Distinguishability Evaluation Factor) was proposed to evaluate the reliabilities of the detection results of two sensors. Then a multi-modal fusion moving objects detection based on F-BDEF was proposed, in which F-BDEF was used to distinguish between false positive alarm and false negative alarm, and to choose the accurate detection region from visible light result and infrared result. The experiments showed that the proposed detection method received more accurate results and could overcome many disturbances, such as sudden change of illumination, shadow, ghost, low-contrast night scene.