Pavement distress classification is a vital step for automatic pavement inspection and maintenance. Recently, patch-based approaches have achieved promising performances and thus extensive attention in this field. However, these methods simply assume that all patches contribute equally to the distress classification, leading to weakly discriminating abilities of models. Moreover, their tedious processes also leads to a low efficiency in inference. In this letter, we present a novel patch-based pavement distress classification approach named Deep Patch Soft Selective Learning (DPS$^2$L), which addresses these issues. Similar to other patch-based approaches, DPS$^2$L partitions the pavement images into patches and aggregates the patch features to accomplish the task. To address the first issue, we introduce a succinct Soft Patch Feature Selection Network (SPFSN) to assess the importance of each patch to the distress classification with a score based on its feature. These scores will be considered as patch-wise weights for feature aggregation. In such a manner, the most discriminative patches are selected in a soft way, and thereby benefit the final classification. To address the inference efficiency issue, knowledge distillation is leveraged to transfer the classification knowledge from DPS$^2$L to the image-based approaches, such as EfficientNet-B3. This distilled model enables incorporating both the advantages of patch-based approaches in classification performance and the advantages of image-based approaches in inference efficiency. Extensive experiments on a large-scale pavement image dataset named CQU-BPDD demonstrates the superiority of our methods over baselines regardless of performance or efficiency.