Maintenance of a desirable mixture of shrubs and grasses is a key issue in sustainable grazing management. The aim of this study was to explore the effect of long-term sheep grazing on desert steppe shrubs. Based on a long-term controlled sheep stocking rate experiment in a desert steppe of north China, combined with long-term random sampling monitoring of above-ground vegetation standing crop (14 years) and short-term systematic sampling monitoring of vegetation cover and individual characteristics of shrubs (3 years), we analyzed plant community changes, the current situation of shrubs and the response of individual shrub characteristics to stocking rate. We found that low stocking rates have increased the cumulative above-ground standing crop of shrubs and herbaceous plants, but the cumulative above-ground standing crop of shrubs under high grazing rates tends to be flat. The cover and height of four shrub species generally showed a decrease with increasing stocking rate, while the response of the four shrubs to the stocking rate gradient varied. Among four shrub species, Artemisia frigida was the most sensitive to stocking rate, followed by Ceratoides latens and Caragana microphylla, while Kochia prostrata was relatively insensitive to stocking rate. These results suggest that grassland managers can use an appropriate stocking rate to maintain desirable plant community composition and configuration in the temperate grassland.