Per-layer optimization in multiuser MIMO systems with Tomlinson-Harashima precoding

In this letter, we investigate the application of Tomlinson-Harashima precoding (THP) in the downlink of multiuser multiple-input multiple-output (MIMO) systems, where multiple antennas are located at all the transceivers. Based on the criterion of maximum system sum-capacity, a per-layer optimization scheme is proposed, in which the subchannel ordering and transceiver filters design are generated. In the proposed scheme, the successive character of THP can be fully exploited, so that both the minimum cost of interference suppression and the maximum power and diversity gains can be implemented, and hence, the system sum-capacity can be improved effectively.