Mass in the immediate environment of a gravitational lens galaxy or projected along the line of sight (LOS) can affect strong lensing observables more than current measurement errors. To quantify the resulting biases and uncertainties in lens model determined quantities like the Hubble constant \(H_{0}\), we consider three lens models that treat the environment and sightline to the lens in different ways: the first ignores mass external to the lens (Lens-Only), the second adds an external shear to the lens plane (Lens+Shear), and the third employs our new framework for multi-plane lensing \citep{McCully14} to build the full three-dimensional mass configuration (3-D Lens), including the additional mass from structures in the lens environment and sightline, as well as foreground and background voids. The Lens-Only model yields poor and biased fits. While the Lens+Shear model can account for tidal stretching from perturbing galaxies at the lens redshift and in the background, it requires corrections for external convergence and cannot fully reproduce the more complicated effects from perturbing foreground galaxies. Critically, our 3-D Lens model, which explicitly includes convergence, recovers lens model parameters without bias and with a scatter driven only by the lens profile degeneracy. For computational efficiency, we quantify which galaxies—by a combination of mass, projected distance from the lens, and offset from the lens redshift—can be treated with the tidal approximation and which need to be treated exactly in the 3-D Lens model. While it not surprising that massive galaxies lying close to the lens are significant perturbers, we also find that foreground structures affect the lens potential more. There is a dramatic variation in the strength of the LOS and environment effects across different lens fields. By modeling each field individually, we produce stronger priors on \(H_{0}\) than by ray-tracing through N-body simulations. We show that lens systems with very asymmetric lens configurations, i.e., those produced by highly elliptical main lens galaxies, are less sensitive to the lens profile degeneracy, thus producing stronger constraints on \(H_{0}\) and making appealing targets for LSST follow-up surveys.