In order to guide the machining of a 600 mm hyperboloid mirror, the detection system of Ronchi grating is designed. In view of the stray light phenomenon in the process of system detection, the causes of the phenomenon are analyzed, and the corresponding solutions are put forward. The analysis results show that when the residual reflection of the splitting prism is greater than 5%, the illumination of the dark fringe on the image plane increases by 27 times, and the contrast of the fringe decreases, making it difficult to distinguish. With the increase of the residual reflection of the grating, bright spots appear in the center of the fringe, and the fringe cannot be distinguished. The background illumination of the bright spot area of the spectroscope is increased by more than 40 times by the striped surface, and the background illumination of the bright spot area of the glass surface is increased by more than 80 times. By optimizing the system structure, the spectroscope and grating are replaced and treated. The stray light phenomenon is effectively solved and the contrast of the image plane fringe is improved.