With the high accuracy of the stellar angular positions, the camera calibration with the night sky observation is available and indispensable. In order to estimate the camera parameters even with higher order nonlinear distortions, a two-step model and the corresponding iterative optimization based on the invariability of inter-star angle were developed in this paper. Furthermore, a compact recursive average filter was designed to improve the accuracy while not adding significantly computation time. The experimental results demonstrated that the proposed model can implement star tracker camera calibration with higher accuracy than the known method.