Fast mode selection for intra and inter prediction

H.264 video coding standard is high complex and time-consuming owing to its multi-mode in intra prediction and multi-size sub-block modes in inter prediction to improve the efficiency of I frame and P frame. A new algorithm with fast mode selection based on wavelet transforms in intra prediction and combined with pre-estimation of intra mode in inter prediction was proposed. Firstly, the method utilized the optimal modes selected in low frequent part transformed by wavelet to estimate the possible modes in sub-block of original macro-block. This method reduces the calculate times from 9 to 3.75 averagely. Secondly, pre-estimation of intra mode in inter prediction was implemented using the parameter of mean removed mean absolute difference, and it reduced the P frame coding time. Simulation experiment results from video sequences show the method effective with the encoding time of sequences lower about 19% averagely while peak signal-to-noise ratio lower 0.0664dB at most.