Accompanied with the development of urbanization and industrialization, China’s energy consumption has been increasing, which has brought about serious environmental problems. Using data about energy consumption, urbanization, economy, population, technology progress of China, this paper established a quantile regression model among energy consumption and these four factors. We also obtained regression equations at different distributions of energy consumption. The results show that there are positive correlations between energy consumption and economic development at all quantiles, so are population and energy consumption. While there are uncertain correlations between energy consumption and technology progress at different quantiles, as well as energy consumption and urbanization. Furthermore, the influential intensity of these factors at different quantile of energy consumption distribution are substantially different. Finally, we will put forward some policy recommendations on energy conservation to help correlative research and practices temperately.