Quantitatively assessing the abundance of the composite minerals of terrestrial granite is crucial for understanding the evolutionary history of the Earth’s crust and for mineral exploration. Prevalent methods based on the Hapke model to estimate mineral abundance by setting the optical constants of endmembers ahead are no longer applicable to terrestrial granite because of the complexity of natural granite, which will lead to remarkable uncertainties in estimations. In this study, we retrieved such specific photometric parameters from the bidirectional reflectance spectra measured at a range of incidence, emergence, and phase angles before they were inputted to the Hapke model to estimate mineral abundance. Four different kinds of granite samples of main rock-forming minerals (quartz, alkali feldspar, and plagioclase) comprising the bulk of granite were used to test the effectiveness of our proposed method. The effects of particle size and dark minerals on the inversion results using the visible near and shortwave infrared (VNIR-SWIR) wavelengths were also examined. The results show that using the photometric parameters retrieved from multiangle measurements as inputs for the Hapke model accurately estimated the abundances of quartz, alkali feldspar, and plagioclase for both natural and synthetic granite samples. Furthermore, the results also prove that the retrieved particle sizes of particulate samples are all within their measured ranges. These results indicate that the proposed approach provides a more accurate and efficient estimation of the compositions of terrestrial granite and is feasible for quickly assessing the abundance of minerals contained in granite.