Nonlinear unmixing can explain the nonlinear mixing effect in complex scenarios of hyperspectral imagery, but the spectral variability of ground objects is one of the difficulties. An unsupervised nonlinear unmixing algorithm dealing with spectral variability is proposed in this paper. The original hyperspectral image data is implicitly mapped into a high-dimensional feature space through a kernel function and then linear unmixing is applied for hyperspectral imagery in combination with spectral variability in this space. Further, local smoothness constraint is added on abundances and coefficients of spectral variability according to the distribution characteristics of ground objects. Experimental results on simulated and real hyperspectral data indicate that, the proposed algorithm can overcome the spectral variability problem in different nonlinear mixing scenarios and improve the unmixing accuracy.