Compared with H.264, the latest High Efficiency Video Coding standard (HEVC) adopts more flexible coding structures and more complex coding modes which significantly brings both high improvement in compression ratio and a huge increase of computational complexity. To reduce the computational complexity, this paper proposes a fast coding unit (CU) size decision algorithm which could effectively improve the speed of intra encoding in HEVC. With statistical analysis for spatial correlation of images, we first classify the CUs into three cases according to different neighboring block partitioning. Then, by referencing the neighboring CUs and combining with necessary comparison of luma variances, the CU size is determined in early. Experimental results show that the proposed improved algorithm can save about 35% intra encoding time with small increase of bitrate and negligible loss in PSNR compared with the high efficiency video coding test model (HM) 12.0 reference software under the same configuration. Therefore the computational complexity can be reduced greatly.