In order to overcome the limit of single-mode approximation(SMA),the dispersion relation was obtained by eigen-function method.The coupling impedance was subsequently derived from the relevant equations.A lot of numerical computations and analysis were done for two typical structures of rectangular waveguide grating: the shallow grating and the deep grating.And the effects of the geometrical dimensions of the grating on high frequency characteristics of system were analyzed.Rectangular gratings models at 3cm band and 8mm band were designed and measured,The experimental values match theoretical values very well.According to linear theory,the "hot" dispersion equations were deduced,and the small-signal gain was achieved.The influences of the radius and current of the electron beam,the acceleration voltage and the geometrical dimensions of the slow-wave structure on the small signal gain were discussed.The results presented in this study provide theoretical basis on designing the rectangular grating traveling wave tube.