Elevated nitrate concentrations in German water bodies are a widespread problem, potentially resulting from a long history of excess nitrogen (N) inputs. Here, we investigated long-term (1950-2014) N dynamics across 89 German catchments using a process-based model. Results showed that the mean fractions of N surplus (excess) exported to the river, removed by denitrification, accumulated in the soil zone, and accumulated in groundwater across all catchments are 27%, 58%, 14%, and 1%, respectively. Dissolved inorganic N in groundwater could affect the stream N levels over decades as indicated by long groundwater transit times. A cluster identified four catchment groups with distinct archetypal long-term N transport and retention dynamics, which can be partly linked to the catchments’ topographic and geological conditions. This hints at underlying mechanisms that explain spatial differences in the fate of diffuse N inputs to catchments and opens the possibility for better-targeted management