Viruses span an impressive size range, with genome length varying a thousandfold and virion volume nearly a millionfold. For cellular organisms the scaling of traits with size is a pervasive influence on ecological processes, but whether size plays a central role in viral ecology is unknown. Here we focus on viruses of aquatic unicellular organisms, which exhibit the greatest known range of virus size. We develop and synthesize theory, and analyze data where available, to consider how size affects the primary components of viral fitness. We argue that larger viruses have fewer offspring per infection and slower contact rates with host cells, but a larger genome tends to increase infection efficiency, broaden host range, and potentially increase attachment success and decrease decay rate. These countervailing selective pressures may explain why a breadth of sizes exist and even coexist when infecting the same host populations. Oligotrophic ecosystems may be enriched in “giant” viruses, because environments with resource-limited phagotrophs at low concentrations may select for broader host range, better control of host metabolism, lower decay rate, and a physical size that mimics bacterial prey. Finally, we describe where further research is needed to understand the ecology and evolution of viral size diversity.