RATIONALE: The multi-reflection time-of-flight mass spectrograph (MRTOF-MS) is a complex nonlinear system with dozens of variables that are impossible to determine in theory. Numerical analysis is the only method to determine a solution. Therefore, a numerical simulation is applied with a modified Nelder–Mead simplex (MNMS) algorithm for optimizing voltage configurations. METHODS: Ion trajectories for injection and confinement are simulated using the software SIMION 8.1. The goal of optimization is to find a more suitable configuration for the electric field. This task becomes more challenging as the number of variables, the complexity of the objective function, and the accuracy of the variable intervals increase. A simplex search algorithm was used to perform the optimization process. We modified the searching algorithm by incorporating a variable transformation to ensure that the variables have smooth boundaries. Additionally, we introduced a dedicated benchmark to facilitate global searches. RESULTS: By iteratively using the MNMS algorithm, a total of eight electrodes have been optimized, resulting in a smaller beam size and more efficient ion transport. CONCLUSIONS: The MNMS algorithm is effectively for optimizing nonlinear MRTOF-MS system. It improves the adaptability and globality of the original algorithm, making it applicable for the numerical analysis of complex mass spectrometry systems and problems in engineering.