Previous models of Mercury's core magnetic field based on high altitude data from first MESSENGER flybys revealed an axisymmetric. Here we use low altitude MESSENGER data covering the entire mission period to construct spherical harmonic models based on various spatial norms. Although we find a dominantly axisymmetric field, our models nevertheless include detectable deviations from axisymmetry. These non-axisymmetric features appear at high latitudes, resembling intense geomagnetic flux patches at Earth's core-mantle boundary. Based on this core field morphology, we then attempt to infer Mercury's internal structure. More specifically, assuming that Mercury's high-latitude non-axisymmetric features are concentrated by downwellings at the edge of the planet's inner core tangent cylinder, and accounting for the presence of a stably stratified layer at the top of Mercury's core, we establish a relation between the inner core size and the thickness of the stratified layer. Considering plausible ranges, we propose that Mercury's inner core size is about 500-660 km, which corresponds to a stratified layer thickness of 880-500 km, respectively.