The present work investigated the mixed crystal structure of multiferroic material Yb0.9Sr0.1MnO3 and studied the influence of nanocrystalline size on its physical properties. The X-ray and neutron diffraction analysis of Yb0.9Sr0.1MnO3 reveals a mixed orthorhombic/hexagonal phase. The average crystallite size observed from XRD and SEM analysis is ~ 70–85 nm. The orthorhombic phase, characterized by a space group Pnma (No. 62), decreases from 45 to 5% with heat treatment, while quantity of the hexagonal phase characterized by a space group P63cm (No.185) increases accordingly. It is found that the lattice parameters of hexagonal crystal system at room temperature are a = b = 6.06479Å, c = 11.42647Å, γ = 120° and the orthorhombic one are a = 5.390Å, b = 7.5440Å and c = 5.430Å. Two antiferromagnetic ordering temperature points of Yb0.9Sr0.1MnO3 are observed near 87 K and 120 K, attributed to C-type antiferromagnetic ordering and Γ2-type antiferromagnetic ordering, respectively. The magnetic moment of the Γ2 antiferromagnetic phase was deduced to be 3.4μB/Mn at 2.5 K. The magnetic moment of the C-type antiferromagnetic phase was found to be 1.2μB/Mn at 2.5 K. A theoretical model describing the magnetization as a function of crystal structure and temperature, based on the Monte Carlo simulation is presented. The internal energy was calculated based on the Ising model, a crucial part of the methodology. The magnetization behavior exhibits a first-order phase transition at low H while a second-order phase transition at high H. Theoretical calculations not only confirmed but also validated the experimental results and their interpretation, providing a solid foundation for the study