Infrared small target detection is a key technology in the field of national defense and security， yet it faces severe challenges posed by complex real-world scenarios. Infrared small targets often present detection challenges such as blurred appearance and low signal-to-clutter ratio due to their extremely low pixel occupancy and interference from complex backgrounds. In recent years， detection methods based on deep convolutional networks have made significant progress； however， most existing approaches focus primarily on spatial feature modeling， overlooking the value of frequency-domain features and the information loss caused by pooling operations. To address these issues， this paper conducts research from a frequency-domain perspective and proposes a novel infrared small target detection network called WDFA-Net， which combines wavelet-aware down sample with multi-frequency cross-attention. The network replaces traditional pooling with a wavelet-aware down sample module （WAD）， enabling dual-branch sampling in both spatial and frequency domains to preserve feature details while reducing dimensionality and enhancing target responses. Furthermore， a multi-frequency cross attention module （MFCA） is designed for feature fusion stage. This module constructs a physically meaningful self-attention mechanism based on multi-frequency components， fully leveraging the complementary information between high- and low-frequency features to achieve inter-layer feature interaction， fusion， and noise suppression. Experimental results demonstrate that WDFA-Net achieves excellent detection performance on the IRSTD-1k， SIRST， and SIRST-UAVB datasets， exhibiting strong robustness in complex real-world scenarios.