The aim of this study is to reduce the losses caused by overtopping-induced dam break. We established a coupled model based on the mechanism of the break process of clay core dam. The hydrodynamic model is a solid-liquid two-phase turbulence equation in consideration of the influence of river bed deformation on water flow. We reconstructed the free surface by adopting PLIC-VOF method, simulated the solid wall boundary using the immersion boundary method based on dispersive force, and judged the slip failure/sliding failure of the core wall by the moment method. We further validated the model by flume experiment, and proved that the model can simulate meticulously and accurately the interaction among the core, the water flow and the bed deformation. The predicted flow rate of the breach is also in good agreement with the measured value. The study can serve as a useful tool for further research on the mechanism of overtopping failure of clay core dams.