Arbuscular mycorrhizal fungi (AMF), which can form symbiotic associations with many terrestrial plants, are critical for crop yields and agroecosystem sustainability. In this study, we assessed the influence of rice–upland crop rotations on soil AMF diversity and composition. We also explored the mechanisms of rice–upland crop rotations that affect AMF using trait-based guild methods. We found that rotations of rice with different plants differentially influenced soil AMF. Rice–wheat (RW) and rice–Chinese milk vetch (RV) rotations significantly altered the soil AMF composition, with RW and RV significantly increasing and decreasing AMF diversity, respectively, compared with the rice–fallow (RF) treatment. In addition, RW and RV affected AMF abundance in intra- and extra-radical portions in different ways. For example, both the RW and RV treatments increased AMF spore density, but decreased AMF colonization rate. Different AMF guilds showed different responses to rice–upland crop rotations. The RW treatment increased the rhizophilic guild by 4.9% and decreased the edaphophilic guild by 27.9%, while the RV treatment produced opposite trends. The rhizophilic and edaphophilic guilds were moderated mainly by soil pH, but the former was also significantly influenced by soil available P and the N:P ratio. Structural equation modeling analysis showed that AMF root abundance (colonization rate) was directly and significantly negatively correlated with rice yield under different rotations. Thus, rice–upland crop rotations changed soil AMF diversity, AMF composition, and trait-based guilds in different ways, and rice yield was mainly correlated with AMF colonization rate.