The concentrations of eight heavy metals (Cr, Hg, As, Pb, Cd, Cu, Zn, Ni) in six river sediment samples were collected for evaluation of the degree of the heavy metals pollution distribution and ecological risk of three main rivers' sediments in Jinan. Multivariate statistical techniques were used to determine the most common pollution sources. The results illustrated that all of the metals in Damatou and Xinfengzhuang sections of the Xiaoqing River were much higher than the background value, and the level of potential ecological risk index was very high. The remaining four sections had a low or moderate degree of ecological risk. Principal component analysis (PCA) showed that all metals, with the exception of As, formed the first component explaining 86.85% of the total variance and industry sources could be considered as the first component, while As alone could be the second component, representing agricultural source. The elements Cr and Zn were grouped together while the remaining six metals formed a separate category. Among all heavy metals, Hg and Cd were the most significant contributors to the pollution. Therefore, the prevention of pollution should pay more attention to controlling the sources, especially Hg and Cd.