Ocean phytoplankton play a critical role in the global carbon cycle, contributing ~50% of global photosynthesis. As planktonic organisms, phytoplankton encounter significant environmental variability as they are advected throughout the ocean. How this variability impacts phytoplankton growth rates and population dynamics remains unclear. Here, we systematically investigated the impact of different rates and magnitudes of sea surface temperature (SST) variability on phytoplankton community growth rates using surface drifter observations from the Southern Ocean (> 30oS) and a phenotype-based ecosystem model. Short-term SST variability (<7 days) had a minimal impact on phytoplankton community growth rates. Moderate SST changes of 3-5oC over 7-21 days produced a large time lag between the temperature change and the biological response. The impact of SST variability on community growth rates was nonlinear and a function of the rate and magnitude of change. Additionally, the nature of variability generated in a Lagrangian reference frame (following trajectories of surface water parcels) was larger than that within an Eulerian reference frame (fixed point), which initiated different phytoplankton responses between the two reference frames. Finally, we found that these dynamics were not captured by the Eppley growth model commonly used in global biogeochemical models and resulted in an overestimation of community growth rates, particularly in dynamic, strong frontal regions of the Southern Ocean. This work demonstrates that the timescale for environmental selection (community replacement) is a critical factor in determining community composition and takes a first step towards including the impact of variability and biological response times into biogeochemical models.