Origins of material in the ocean are commonly identified by tracing Lagrangian particle trajectories backward-in-time in two or three dimensions. While this is mathematically consistent, numerical computations are hampered by numerical round-off and truncation errors, leading to discrepancies between forward- and backward-in-time trajectories. The chaotic nature of ocean flows amplifies these errors. We identify an additional issue with Lagrangian backtracking, related to the reversal of stability with regards to velocity convergence and divergence. Trajectories near convergent regions are stable to numerical errors when calculated forward-in-time but become unstable backward-in-time. The timescales at which trajectories reside in convergent zones are thus underestimated in backward-in-time computations, meaning convergent regions (downwelling zones) become underrepresented and divergent zones (upwelling zones, river mouths) overrepresented as trajectory sources. Using mesoscale experiments representing common set-ups, we show that already for timescales of less than half a year, this leads to systematic biases in the regions identified as particle origins. These biases can extend over distances of thousands of kilometers. While this stability bias is linked to divergence, it is not only limited to 2D trajectories in 3D flows, as we discuss how inappropriate treatment of surface boundary conditions in 3D Lagrangian studies can also introduce an effective non-zero divergence. These findings have consequences for source-attribution modeling, for example in the context of water mass tracing, ecology, and pollution studies. Backtracking is typically applied to material that has accumulated in convergent zones, where the stability bias is especially relevant, which further impedes source attribution.