Mesoscale convective systems (MCSs) are clusters of thunderstorms that are important in Earth’s water and energy cycle. Additionally, they are responsible for extreme events such as large hail, strong winds, and extreme precipitation. Automated object-based analyses that track MCSs have become popular since they allow us to identify and follow MCSs over their entire life cycle in a Lagrangian framework. This rise in popularity was accompanied by an increasing number of MCS tracking algorithms, however, little is known about how sensitive analyses are concerning the MCS tracker formulation. Here, we assess differences between six MCS tracking algorithms on South American MCS characteristics and evaluating MCSs in kilometer-scale simulations with observational-based MCSs over three years. All trackers are run with a common set of MCS classification criteria to isolate tracker formulation differences. The tracker formulation substantially impacts MCS characteristics such as frequency, size, duration, and contribution to total precipitation. The evaluation of simulated MCS characteristics is less sensitive to the tracker formulation and all trackers agree that the model can capture MCS characteristics well across different South American climate zones. Dominant sources of uncertainty are the segmentation of cloud systems and the treatment of splitting and merging of storms in MCS trackers. Our results highlight that comparing MCS analyses that use different tracking algorithms is challenging. We provide general guidelines on how MCS characteristics compare between trackers to facilitate a more robust assessment of MCS statistics in future studies.