Spectral unmixing for hyperspectral remote sensing images is always required due to the existence of mixed pixels. However, most spectral unmixing algorithms at present are proposed based on the linear mixture model which may be invalid in many real-world scenarios with nonlinear spectral mixing effects. Therefore, nonlinear mixture models and their corresponding algorithms should be considered to improve the accuracy of endmember extraction and abundance estimation. This paper aims to introduce the recent advances in nonlinear spectral unmixing models and algorithms focusing on two main typical nonlinear mixing scenarios: intimate mineral mixtures and multilayer mixtures in vegetation covered areas. Further, data-driven nonlinear spectral unmixing algorithms such as kernel methods and manifold learning are also presented here. Finally, both advantages and defects of these models and algorithms are summarized and future research trends are analyzed.