Because of the explosive growth of network traffic, backbone networks move towards all-optical-packet-switching networks. However, optical buffers implemented by Fibre Delay Lines (FDLs) have a volatile nature because of signal loss and noise accumulation problems from optical components of the FDL buffer. Because of this, scheduling packets becomes more difficult in the FDL buffer than in the RAM buffer, and requires additional design considerations for reducing packet loss. Therefore, we propose a packet prioritisation algorithm for all-optical-packet-switching networks with FDL buffers. The point of the packet prioritisation algorithm is to rank packets in optimal balance between latency and residual distance. We optimise a relative-distance factor to minimise packet loss probability. Our simulation results show how various network parameters affect the optimal relative-latency factor.