A modified algorithm of marker-based watershed segmentation was proposed by combining spectral similarity with phase congruency model in this paper. The performance of segmentation using marker-based watershed algorithm was decided by the result of edge detection from remotely sensed imagery. Thus we use spectral similarity of the same type ground object from remotely sensed imagery to suppress fake edges and noises to retrieve good segmentation results. In this paper, a spectral similarity model defined by the sum of distance of spectral curve between the target pixel and adjacent pixels was introduced into phase congruency model for edge detection. Then segmentation of remotely sensed imagery was obtained by using auto marker-based watershed algorithm. Finally, an unsupervised evaluation and comparison of the image segmentation from the proposed algorithm and some other existing algorithms was implemented using information entropy. Furthermore, the computation time of the proposed algorithm was also compared with other algorithms. The experimental segmentation results show that the proposed algorithm can reduce the over-segmentation phenomenon efficiently and it is readily to obtain better segmentation results by using this algorithm.