We consider the design of a hybrid multiple-input multiple-output (MIMO) processor consisting of a radio frequency (RF) beamformer and a baseband MIMO processor for millimeter-wave communicationsover multiuser interference channels. Sparse approximation problems are formulated to design hybrid MIMO processors approximating the minimum-mean-square-error transmit/receive processors in MIMO interference channels.
They are solved by orthogonal-matching-pursuit-based algorithms that successively select RF beamforming vectors from a set of candidate vectors and optimize the corresponding baseband processor in the least squares sense. It is shown that various beamformers can be designed by considering different types of candidate vector sets. Simulation results demonstrate the advantage of the proposed design over the conventional method that designs the baseband processor after steering the RF beams.