Forecasting the arrival time of Earth-directed coronal mass ejections (CMEs) via physics-based simulations is an essential but challenging task in space weather research due to the complexity of the underlying physics and limited remote and in-situ observations of these events. Data assimilation (DA) techniques can assist in constraining free model parameters and reduce the uncertainty in subsequent model predictions. In this study, we show that CME simulations conducted with the Space Weather Modeling Framework (SWMF) can be assimilated with SOHO LASCO white-light (WL) observations and solar wind observations at L1 prior to the CME eruption to improve the prediction of CME arrival time. The L1 observations are used to constrain the model of the solar wind background into which the CME is launched. Average speed of CME shock front over propagation angles are extracted from both synthetic WL images from the Alfv\’en Wave Solar atmosphere Model (AWSoM) and the WL observations. We observe a strong rank correlation between the average WL speed and CME arrival time, with the Spearman’s rank correlation coefficients larger than 0.90 for three events occurring during different phases of the solar cycle. This enables us to develop a Bayesian framework to filter ensemble simulations using WL observations, which is found to reduce the mean absolute error of CME arrival time prediction from about 13.4$\pm$3.8 hours to 5.1$\pm$3.0 hours. The results show the potential of assimilating readily available L1 and WL observations within hours of the CME eruption to construct optimal ensembles of Sun-to-Earth CME simulations.