Spatial patterns of neutral genetic diversity are often investigated to infer gene flow in wild populations. However, teasing apart the influence of gene flow from the effect of genetic drift is challenging given that both forces are acting simultaneously on patterns of genetic differentiation. Here, we tested the relevance of a distance-based metric -based on estimates of effective population sizes or on environmental proxies for local carrying capacities- to assess the unique contribution of genetic drift on pairwise measures of genetic differentiation. Using simulations under various models of population genetics, we demonstrated that one of three metrics we tested was particularly promising: it correctly and uniquely captured variance in genetic differentiation that was due to genetic drift when this process was modelled. We further showed that (i) the unique contribution of genetic drift on genetic differentiation was high (up to 20 %) even when gene flow was high and for relatively high effective population sizes, and (ii) that this metric was robust to uncertainty in the estimation of local effective population size (or proxies for carrying capacity). Finally, using an empirical dataset on a freshwater fish (Gobio occitaniae), we demonstrated the usefulness of this metric to quantify the relative contribution of genetic drift and gene flow in explaining pattern of genetic differentiation in this species. We conclude that considering Isolation-by-Drift metrics will substantially improve the understanding of evolutionary drivers of observed spatial patterns of genetic variation.