Reforestation and check dam construction have been progressively implemented on the Loess Plateau for several decades. However, it is still unclear how the two major sediment control strategies affect soil erosion and sediment yield in a large watershed. A combination of field investigation and model simulation was employed to quantitatively identify the impacts of the two measures on soil erosion and sediment yield in the Kuye River watershed. Significant land use changes, with the conversion of arable land and bare land to vegetation cover and construction land, occurred in the study watershed from 1987 to 2016. In addition, 306 key dams were built in the watershed, with a total storage capacity of 316.64 Mm3, according to the statistical data of 2011. Hot spot analysis showed that the high-risk regions for soil erosion and sediment yield were mainly concentrated on the periphery of Shenmu County and the outlet of the watershed. The simulation results showed that the land use changes from 1987 to 2016 remarkably reduced sediment yield by 51.14% without considering the action of check dams. In the 1987 scenario, the sediment yield was reduced by 50.44% when considering the action of check dams compared with the yield that was estimated without consideration of check dams. Under the combined effect of the two factors, the sediment yield decreased by 73.91% in 2016. More attention should be paid to check dams, and corresponding measures should be taken to protect them, especially in the flood period.