Early life adversity predicts shorter adult lifespan in several animal taxa. Yet, work on long-lived primate populations suggests the evolution of mechanisms that contribute to resiliency and long lives despite early life insults. Here, we tested associations between individual and cumulative early life adversity and lifespan on rhesus macaques at the Cayo Santiago Biological Field Station using 50 years of demographic data. We performed sex-specific survival analyses at different life stages to contrast short-term effects of adversity (i.e., infant survival) with long-term effects (i.e., adult survival). Rhesus macaques exposed to adversity at birth suffered a significant increase in mortality risk during infancy with both individual and cumulative adversities having the highest impact among affected females. However, when considering adult lifespan, affected males showed higher vulnerability to both individual and cumulative adversities early in life. Our study shows profound immediate effects of insults at birth on female infant cohorts and suggests that affected female adults are more robust (i.e., viability selection). In contrast, adult males who experienced harsh conditions early in life showed an increased mortality risk at older ages as expected from hypotheses of long-term effects of individual, as well as cumulative, adversity early in life. Our study reveals that mortality risk during infancy is mainly driven by the type of adversity, rather than their accumulation at birth. However, cumulative adversity seems to play a major role in adult survival. Our analysis suggests sex-specific selection pressures on life histories and highlights the need for studies addressing the effects of early life adversity across multiple life stages. This information is critical for planning life stage-specific strategies of conservation interventions.