\justifying Deep learning has established itself as the go-to methodology for automated event discrimination. However, these methodologies still provide suboptimal performance in many classification tasks, including the seismic source categorization problem considered in this article. Here, we develop a novel deep learning framework that allows for the direct integration of environmental context, which we refer to as Physically-Augmented Deep Learning (PADL). Specifically, we augment the learning process by incorporating seismic velocity models generated from a physics-based simulator. Our experiments couple real observational waveform data and synthetic velocity models from the Tularosa Basin region and demonstrate near-perfect classification accuracy when employing PADL. A robust set of ablation studies on joint and independent convolutional neural networks and various combinations of real and simulated input data confirm the efficacy of our PADL framework.