A Variational Approach to Small-Scale Parameterization for Nonlinear and Stochastic Dynamical Systems