Effects of physical and morphometric factors on nutrient removal properties were studied in small agricultural ponds with different depths, volumes, and residence times in western Japan. Average residence time was estimated to be >15 days, and it tended to decrease from summer to winter because of the increase in water withdrawal for agricultural activity. Water temperature was clearly different between the surface and bottom layers; this indicates that thermal stratification occurred in summer. Chlorophyll-a was significantly high (>20 μg/L) in the surface layer (<0.5 m) and influenced by the thermal stratification. Removal ratios of dissolved total nitrogen (DTN) and dissolved total phosphorus in the ponds were estimated to be 53–98% and 39–98% in August and 10–92% and 36–57% in December, respectively. Residence time of the ponds was longer in August than in December, and DTN removal, in particular, was more significant in ponds with longer residence time. Our results suggest residence time is an important factor for nitrogen removal in small agricultural ponds as well as large lakes.