The dynamics of virulence evolution in vector-born plant pathogens can be complex. Here we use individual-based simulations to investigate how virulence evolution depends on genetic trade-offs and population structure in pathogen populations. Although quite generic, the model is inspired by the ecology of the plant-pathogenic bacterium Xylella fastidiosa, and we use it to gain insights into possible modes of evolution of virulence in that group. In particular, we aim to sharpen our intuition about how virulence may evolve over short time scales in response to decreases in vector efficacy. We find that even when pathogens find themselves much more often in hosts than vectors, selection in the vector environment can cause correlational and potentially non-adaptive changes in virulence in the host. The extent on such correlational virulence evolution depends on many system parameters, including the pathogen transmission rate, the relative proportions of the pathogen population occurring in hosts versus vectors, the strengths of selection in host and vector environments, and the extent of virulence per se. But there is a statistical interaction between the strength of selection in vectors and the predominance of pathogens in hosts, such that if within-vector selection is strong enough, the predominance of pathogens within hosts has little effect on the evolution of virulence.