The Yellowstone magmatic system is one of the largest magmatic systems on Earth, and thus an ideal location to study magmatic processes. Whereas previous seismic tomography results could only image a shallow magma reservoir, a recent study using more seismometers showed that a second and massive partially molten mush reservoir exists above the Moho \citep{huang2015yellowstone}. To understand the measurable surface response of this system to visco-elasto-plastic deformation, it is thus important to take the whole system from the mantle plume up to the shallow magma reservoirs into account. Here, we employ lithospheric-scale 3D visco-elasto-plastic geodynamic models to test the influence of parameters such as the connectivity of the reservoirs and rheology of the lithosphere on the dynamics of the system. A gravity inversion is used to constrain the effective density of the magma reservoirs, and an adjoint modelling approach reveals the key model parameters affecting the surface velocity. Model results show that a combination of connected reservoirs with plastic rheology can explain the recorded slow vertical surface uplift rates of around 1.2 cm/yr, as representing a long term background signal. A geodynamic inversion to fit the model to observed GPS surface velocities reveals that the magnitude of surface uplift varies strongly with the viscosity difference between the reservoirs and the crust. Even though stress directions have not been used as inversion parameters, modelled stress orientations are consistent with observations. However, phases of larger uplift velocities can also result from magma reservoir inflation which is a short term effect. We consider two approaches: 1) overpressure in the magma reservoir in the asthenosphere and 2) inflation of the uppermost reservoir prescribed by an internal kinematic boundary condition. We demonstrate that the asthenosphere inflation has a smaller effect on the surface velocities in comparison with the uppermost reservoir inflation. We show that the pure buoyant uplift of magma bodies in combination with magma reservoir inflation can explain (varying) observed uplift rates at the example of the Yellowstone volcanic system.