Results of development of the nonlinear theory of relativistic TWT and BWO on irregular hollow waveguides with cathode filter-modulator with the account as propagating, and beyond cut-off waves are given. The results of optimization on efficiency of variants relativistic Cherenkov oscillators are given. The rigorous theory of excitation of an irregular electrodynamic system is used to solve the problem of determining the profile of output horn of Cherenkov oscillators that provides a required mode structure at the output and an optimum directivity pattern.