Dioecious trees are important components of many forest ecosystems. Outcrossing advantage and sexual dimorphism are two major mechanisms that explain the persistence of dioecious plants; however, they have rarely been studied in dioecious trees. In this study, we investigated the influence of gender and genetic distance between parental trees (GDPT) on the growth and functional traits of 229 seedlings of a dioecious tree, Diospyros morrisiana. We found significant positive relationships between GDPT and seedling sizes and tissue density. However, the positive outcrossing effects on seedling growth mainly manifested in female seedlings, but were not prominent in male seedlings. Male seedlings generally had higher biomass and leaf area than female seedlings, but such differences diminished as the GDPT increased. Our research highlights that outcrossing advantage in plants can be sex-specific and sexual dimorphism begins from the seedling stage of dioecious trees.