Agriculture is increasingly threatened by climate change impacts, especially in lower-income nations where traditional paper survey-based data collection for detailed agricultural reporting can be impractical. Leveraging advancements in remote sensing and machine learning, we propose a Convolutional Neural Network-Long Short-Term Memory model designed to infer farm-level crop yields from one year of high-resolution weather data. We also propose assessing the potential of predicting based on forecasted weather data and implementing transfer learning to predict crop yield in regions outside the US with sparse data collection. If successful, this approach would offer a detailed and adaptive tool for countries with all levels of monitoring resources.