Water distribution networks (WDNs) play a crucial role in the well-being of human populations and economic prosperity. It is essential that they cope with abnormal operating conditions and recover functionality quickly. Traditionally, WDNs are designed using cost and reliability objectives, but there is a lack of consensus on the definition and quantification of reliability which typically is a computationally intense process. Subsequently, various reliability-like metrics, called resilience indices, have been developed and demonstrated in the design of WDNs. Few studies exist that thoroughly evaluate the performance of the previously developed resilience metrics. This paper investigates three resilience metrics by evaluating their performance on three benchmark WDNs for several simulated mechanical failure states. The metrics studied are: (a) resilience index, (b) network resilience index (NRI), and (c) modified resilience index (MRI). The metric MRI performed better overall but NRI produced cheaper designs that performed better in the case of WDN-I. It is recommended that a better metric that incorporates different dimensions of resilience, such as robustness and redundancy, should be developed in the future.