Unmanned aerial vehicle (UAV) detection holds significant value in both civilian and military domains, however, conventional infrared detection systems remain vulnerable to background clutter interference. Infrared polarization imaging technology offers a novel solution by integrating polarization data with infrared imaging. However, the differences between polarization and infrared images introduces new problems to target extraction. Therefore, we propose a new detection algorithm based on scale-adaptive local extreme measure (ALEM). The algorithm introduces an enhanced SUSAN operator to quickly extract regions of interest (ROIs) while estimating potential target scales within these regions. Then present the ALEM algorithm, which is specifically designed to exploit the unique characteristics of polarization images. The algorithm effectively measures contrast by analyzing pixel neighborhood features within polarization images. Experimental results based on real-world polarization image dataset demonstrate that: the signal-to-noise ratio gain of the algorithm is increased by 2.7 times, the background suppression factor is increased by 8.6 times, and it can run at 20 fps. It exhibits excellent detection performance, robustness, and the capability for real-time detection.