Understanding the environmental drivers and the genetic architecture of traits involved in local adaptation is of fundamental interest as it may give insight to ecological speciation. Unfortunately, local adaptation studies rarely consider early life stages. As a result, we know surprisingly little about the genetic architecture and evolution of seed-based life history traits especially considering adaptation to xeric and mesic habitats. Here, we evaluate the genetic architecture of seed trait divergence in a recombinant inbred population derived from xeric and mesic ecotypes of Panicum hallii. We found the majority of seed related QTL had overlapping intervals, suggesting tight linkage and possibly pleiotropy are important in seed trait evolution. Our results suggest that the evolution of these traits is non-independent and may involve tradeoffs that affect the direction of divergence. Seed size and germination rate shared two colocalized QTL, each with antagonistic additive effects. This supports the hypothesis of a functional genetic trade-off between these two traits, resulting in either large seed/high dormancy or small seed/low dormancy trait combinations. We identified several candidate genes that may explain this trade-off. Overall, our study provides insights into the factors facilitating and potentially constraining ecotypic differentiation in plants through seed-based life history traits.