Ocean phytoplankton play a critical role in the global carbon cycle, contributing ~50% of global photosynthesis. As planktonic organisms, phytoplankton encounter significant environmental variability as they are advected horizontally across the upper ocean. The impact of this variability on phytoplankton growth rates has not been quantified and is not captured by many current biogeochemical models. Here, we systematically investigated the impact of different rates and magnitudes of sea surface temperature (SST) variability on phytoplankton community growth rates using surface drifter observations from the Southern Ocean (>30°S) and a phenotype-based ecosystem model. Moderate SST changes of 3-5°C over 7-21 days (~4-13 generations for a typical growth rate of 0.5 day-1) produced the largest time lag between the temperature change and the biological response. Shorter term SST variability (<7 days) had little impact on the phytoplankton community growth rates. The impact of SST variability was not captured by the Q10-based model of community growth leading to an overestimation of community growth rates, particularly in dynamic, strong frontal regions of the Southern Ocean. Furthermore, we demonstrated that the nature of variability encountered in a Lagrangian reference frame (following trajectories of surface water parcels) differed from that within an Eulerian reference frame, which resulted in significant effects on phytoplankton dynamics. Our results quantify the temporal scales of SST variability relevant for phytoplankton in the Southern Ocean and take a first step towards including the impact of variability and biological response times into numerical models.