Color in flowers and fruits carries multiple functions, from attracting animal partners (pollinators, dispersers) to mitigating abiotic environmental stress (cold, drought UV-B) through changes in pigment production. However, with research primarily focusing on biotic interactions as selective agents, it remains unclear whether flower and fruit colors underlie the same abiotic stressors and vary jointly across large spatial scales. In light of climate change and biodiversity loss, it is high time to evaluate global color-environment associations. Leveraging a dataset of 2815 animal-pollinated and animal-dispersed species from 51 plant clades, we here show that the distribution and diversity of flower and fruit colors is globally decoupled and shaped by different factors. Fruit color diversity is generally higher and peaks in tropical areas where the diversity of animal mutualists is highest. Flower color diversity, in contrast, peaks in areas of high abiotic environmental stress, with distinct flower colors associating with distinct abiotic factors.