In the complex and changeable sea environment, when using infrared imaging technology to search and rescue small and medium targets on the sea surface, it is necessary to classify the collected original images in order to facilitate the subsequent target processing in different scenes. According to different environmental conditions, the sea infrared images are divided into five kinds of scenes. The training set images are extracted from two aspects: one is to divide an image into basic layer and detail layer by the Gaussian filter, and use improved histogram of oriented gradient (HOG) method to extract the features; the other is to extract features by calculating local contrast of images. The extracted feature vectors are fused and input into the classifier, and the test set images are classified by support vector machine (SVM). In this paper, a new feature descriptor combined with HOG and local contrast method (LCM) is used to classify the scene of sea infrared image. Compared with other methods, the results show that the accuracy of the improved method is 96.4%, which reflects the feasibility and effectiveness.