Artificial barriers cause widespread impacts on freshwater fish. Swimming performance is often used as the key metric in assessing fishes’ responses to river barriers. However, barrier mitigation is generally based on the swimming ability of salmonids and other strong swimmers because knowledge of swimming ability for most other freshwater fish is poor. Also, fish pass designs tend to adopt a ‘one size fits all’ approach because little is known about population or individual variability in swimming performance. Here, we assessed interspecific and intraspecific differences in the sustained swimming speed (Usus) of five freshwater fish with contrasting body sizes, morphologies and swimming modes: topmouth gudgeon, European minnow, stone loach, bullhead and brown trout. Significant Usus variation was identified at three organizational levels: species, populations and individual. Interspecific differences in Usus were as large as 64 cm s−1, upstream populations of brown trout showed mean Usus 27 cm s−1 higher than downstream populations, and species exhibited high individual variation (e.g. cv = 62% in European minnow). Sustained swimming speed (Usus) increased significantly with body size in topmouth gudgeon, European minnow and brown trout, but not in the two benthic species, bullhead and stone loach. Aerobic scope had a significant positive effect on Usus in European minnow, stone loach and brown trout. Sustained swimming speed (Usus) decreased with relative pectoral fin length in European minnow and brown trout, whereas body fineness was the best predictor in stone loach and bullhead. Hence, swimming performance correlated with a diverse range of traits that are rarely considered when predicting fish passage. Our study highlights the dangers of using species’ average swimming speeds and illustrates why a ‘one size fits all’ approach often fails to mitigate for barrier effects. We call for an evidence-based approach to barrier mitigation, one that recognizes natural variability at multiple hierarchical levels.