In conventional terrain classification for the polarimetric SAR (PolSAR) images, color features are rarely involved unless in one recent supervised work. Unlike that work, the color features are exploited in a novel framework for the unsupervised classification of urban areas in this paper. Firstly, based on the recent four-component decomposition model of the PolSAR data, the common color spaces, such as YUV, RGB, HSI, and CIELab are calculated. The color feature is quantitatively selected from these color spaces by introducing the color entropy. Then together with the texture feature and the extended scattering power entropy, the adaptive mean-shift algorithm is used to segment the PolSAR data into clusters. Finally, the clusters are merged according to the G0 distribution-based distance measurement. The proposed framework is verified by the experiments on one AIRSAR L-band and two Radarsat-2 C-band PolSAR data. The classification accuracy indicates that the proposed method has superior discriminative ability for urban areas compared with existing works.