We present a new approach that allows for the inversion of quantities derived from the observed data using non-diagonal data covariance matrices. For example, we can invert approximations of apparent resistivity and phase instead of magnetotelluric impedance using this methodology. Compared to the direct inversion of these derived quantities, the proposed methodology has two advantages: i) If an inversion algorithm allows for the specification of a full data covariance matrix, users can invert for arbitrary derived quantities by specifying the appropriate covariance matrix instead of having to rely on the inversion code to have implemented this feature. ii) It is fully compatible with the assumptions of least-squares optimization and thus avoids potential issues with bias when inverting quantities that are non-linear functions of the original data, We discuss the theory of this approach and show an example using magnetotelluric data. However, the same method can be applied to other types of geophysical data, for example gravity gradient measurements.