An adaptive RQT mode selection algorithm for HEVC

High Efficiency Video Coding (HEVC) is a new video coding standard which is currently under development. In order to achieve better coding efficiency than H.264/AVC High Profile, HEVC adopts new techniques, like residual quad-tree (RQT) coding, to improve the coding efficiency, which is however at the cost of a much higher computational complexity, especially for inter prediction residual coding. Thus it is quite necessary to optimize the RQT mode selection algorithm. In this paper, aiming at achieving better performance/complexity trade-offs, we propose an adaptive RQT mode selection algorithm. We observe, through extensive experiments, that the impact of maximum inter RQT depth on coding performance for different coding unit (CU) sizes are quite different. More specifically, with an increasing CU size, the impact of maximum inter RQT depth on coding performance will decrease. Therefore, we set the maximum inter RQT depth of 64×64 size CU to 1 while set the maximum inter RQT depth of 8×8 size CU to 2 to get better performance/complexity trade-offs. Furthermore, for prediction unit (PU) partition mode that is not 2N×2N, we set its maximum inter RQT depth to 1 and adopt implicit transform unit (TU) partition method. For PU partition mode that is 2N×2N, we extract the information of residual and apply the discriminant analysis method of statistics to select the best RQT mode. Compared to HM 4.0, our proposed algorithm obtains better trade-off between coding performance and computational complexity.