Long-term ecological studies have consistently reported slower than expected changes in biodiversity over time. One explanation for this phenomenon is that commonly used diversity measurements such as species richness are too coarse to detect mechanisms shaping community assembly. It remains unclear whether abundance based diversity measurements are susceptible to the same problem. To test this, we study temporal changes in abundance based diversity indices across 3341 observations from 880 plots from 15 long-term vegetation plot studies. We then partition diversity change into mechanisms of interest to ecologists: selection, drift, and immigration. We show that these mechanisms are an imperfect predictor of temporal diversity change, creating a mismatch between changes in species abundances and changes in diversity, particularly when shifts in diversity are rapid. To resolve this mismatch, we quantify a less studied mechanism “rarity shifts”, which measure how an individual’s contributions to diversity changes over time. We found rarity shifts are an important component of diversity change across many studies. Furthermore, rarity shifts tend to oppose other mechanisms, particularly selection. Therefore, rarity shifts obscure changes in relative abundance from abundance based diversity measurements, revealing why diversity changes appear slower than expected. Ultimately, understanding rarity shifts can lead to a more accurate understanding of the rate and nature of temporal diversity change in ecology and conservation.