Relative to the single-cavity gyrotrons operating in high-order mode, the coaxial cavity gyrotrons have the advantages of reducing mode competition, improving the stability of single mode operation and increasing the power capacity. Therefore, the coaxial cavity gyrotrons are more suitable for electron cyclotron resonance heating and electron cyclotron current driving in controlled thermonuclear fusion and attract much attention. The effects of structure parameters, electron beam parameters and ohmic losses on the beam-wave interaction of a coaxial cavity gyrotron operating at 170 GHz, mode were investigated in detail. Firstly, the mode selection of 170-GHz MW-class gyrotrons was analyzed and mode was chosen as the operating mode. Secondly, based on the time-dependent self-consistent nonlinear theory, a time-domain single-mode steady-state code was written to study the beam-wave interaction. The influences of the beam current, the magnetic field and the ohmic losses on the cavity walls were analyzed and the operating parameters were optimized. The simulation results show that when the voltage, the beam current and the axial guiding magnetic field are designed to be 65 kV, 68 A and 6.58 T, an output with 2.18 MW power and 49.23 % efficiency can be obtained, the peak ohmic loss density on the outer wall of the cavity is 1.94 kW/cm², and the peak ohmic loss density on the insert is less than 0.15 W/cm². The interaction efficiency decreases with the increase of electron velocity spread, and the output frequency shifts downward. The thickness of electron beam has similar effects on the interaction.