This paper proposes an image-based visibility estimation method with deep label distribution learning. To train an accurate model for visibility estimation, it is important to obtain the precise ground truth for every image. However, the ground-truth visibility is difficult to be labeled due to its high ambiguity. To solve this problem, we associate a label distribution to each image. The label distribution contains all the possible visibilities with their probabilities. To learn from such annotation, we employ a CNN-RNN model for visibility-aware feature extraction and a conditional probability neural network for distribution prediction. Our experiment shows that labeling the image with visibility distribution can not only overcome the inaccurate annotation problem, but also boost the learning performance without the increase of training examples.