Sediment transport modeling (STM) is a potentially effective tool for estimating the magnitude of tsunamis and earthquakes without historical records. However, applying STM to prehistorical tsunamis is challenging because of many uncertainties in topography and roughness. In the coast of Hidaka, Hokkaido, Japan, there is potential to conduct STM even in the absence of historical records because of the comprehensive geological data that reveal the coastal evolution during the Holocene in addition to tsunami sediment surveys. The tsunami deposits in Hokkaido suggest the presence of events on a larger scale than historical tsunamis; particularly the 17th-century tsunami had multiple potential wave sources other than a Kuril Trench earthquake, inhibiting its magnitude estimation. In this study, we applied STM to paleotsunamis in the coast of Hidaka, where the wave source is unknown and there are comprehensive geological data. The modeling parameters—paleotopography, roughness, grain size, initial sand source, sea level, and beach ridge height—were estimated using data obtained from geological surveys and sensitivity tests. The modeling of a tsunami induced by a Kuril Trench earthquake reproduced the sediment distributions and sedimentary structures of the observed sand layers better than that of the extreme storm and volcanic tsunami. The paring down wave sources of the sand layer implies that a wider rupture zone in the Kuril Trench is less likely. This case study provides information on the parameters that geologists and modelers should consider when applying STM to paleotsunamis.