As one of the most important video compression technologies, inter prediction coding is highly efficient in reducing the temporal redundancy of video sequence. However, in the latest video coding standard High Efficiency Video Coding standard (HEVC), complicated inter prediction brings high computational complexity and seriously restricts encoding speed. To save encoding time, in this paper, a fast inter prediction algorithm based on Rate-Distortion cost is proposed by selecting different thresholds for coding units(CU) in different depths and through threshold comparison to early terminate the splitting of CU. First, the splitting of largest coding unit (LCU) in depth 0 is determined by the RD cost of its reference picture with best CU size being 64x64. If the LCU is split into sub-CUs, for the CU with depth level from 1 to 2, its splitting is determined by the RD cost of encoded CUs in the same Coding Tree Unit (CTU) and the same depth. With terminating the splitting of CUs early, encoding speed is improved. Experimental results show that the proposed improved algorithm can save about 30% encoding time on the basis of ensuring visual quality and compression ratio of videos. Therefore, the computational complexity can be reduced greatly.