Recent satellite altimeter retracking and filtering methods have considerably reduced the noise level in estimates of the significant wave height (Hs), allowing to study processes with smaller spatial scales. In particular, previous studies have shown that wave-current interactions may explain most of the variability of Hs at scales 20 to 100 km. As the spatial scale of the measurement is reduced, random fluctuations emerge that should be associated to wave groups. Here we quantify the magnitude of this effect, and the contribution of wave groups to the uncertainty in Hs measurements by altimeters, with a particular focus on extreme extra-tropical storms. We take advantage of the low orbit altitude of the China-France Ocean Satellite (CFOSAT), and the low noise level of the nadir beam of the SWIM instrument. Our estimate of wave group effects uses directional wave spectra measured by off-nadir beams on SWIM and signal processing theory that gives statistical properties of the wave envelope, and thus the local wave heights, from the shape of the wave spectrum. We find that the standard deviation of Hs associated to wave groups is a function of satellite altitude, wave height and spectral bandwidth. For CFOSAT these fluctuations generally account for about 25% of the variance measured over a 80 km distance. This fraction is largest in storms and in the presence of long swells. When the estimated effect of wave groups is subtracted from the measured Hs variance, the remaining variability is higher in regions of strong currents.