The development of atmospheric parameterizations based on neural
networks is often hampered by numerical instability issues. Previous
attempts to replicate these issues in a toy model have proven
ineffective. We introduce a new toy model for atmospheric dynamics,
which consists in an extension of the Lorenz'63 model to a higher
dimension. While neural networks trained on a single orbit can easily
reproduce the dynamics of the Lorenz'63 model, they fail to reproduce
the dynamics of the new toy model, leading to unstable trajectories.
Instabilities become more frequent as the dimension of the new model
increases, but are found to occur even in very low dimension. Training
the neural network on a different learning sample, based on Latin
Hypercube Sampling, solves the instability issue. Our results suggest
that the design of the learning sample can significantly influence the
stability of dynamical systems driven by neural networks.