A small infrared target detection algorithm which combines target characteristics with class prediction of local background is proposed. The elimination of false alarms in the detection of small infrared targets in sky is studied in detail. In complex infrared scenes, the false alarms caused by complex and changing ground objects may seriously affect the sensitivity and robustness of a detection system. If the target characteristics are used alone, the false alarms caused by ground objects can be filtered difficultly. Firstly, the latent targets are extracted by using a new Top-Hat transform. Secondly, for each of the latent targets, the likelihood of being true targets is obtained from the target characteristics on the one hand, and another likelihood of true targets is obtained from the prediction of the class label (sky or ground) of the neighboring background on the other hand. Finally, both likelihoods are combined to eliminate the false targets. The experimental results show that compared with the algorithm which uses target characteristics alone, the detection performance of the proposed algorithm is improved greatly.