According to the feature that the resolution of the Acousto-Optical Tunable Filter (AOTF) spectral measurement system changes with the diffraction wavelength, a method using deconvolution algorithm to improve the spectral resolution capability of AOTF spectrometers is studied. Combining with the distribution of spectral resolution, nonlinear coordination transformation is performed in the spectral range so as to keep a same transfer function over the entire spectral range. Then, the nonlinear iterative deconvolution method and Fourier self-deconvolution are used to process the line spectral data acquired by the Fourier spectral system for resolution improvement. The simulation results are restored in resolution to a great extent and the original low resolution information which is lost because of low resolution is recovered.