[Objective] To address the insufficient understanding of the mechanical contact behavior at the interface between a cut-off wall and a clay core wall in earth-rockfill dams, this study aims to investigate the nonlinear normal contact relationship developed during the penetration of a cut-off wall into a clay core wall. The mechanical interaction between the cut-off wall and the core wall is a key factor affecting the deformation compatibility and overall stability of seepage-control systems in earth-rockfill dams. However, the wall-end resistance mobilization mechanism and its dependence on overburden pressure have not been fully clarified. Therefore, this study focuses on establishing an experimentally validated nonlinear normal contact model that can describe the relationship between wall-end pressure and penetration displacement, and further evaluates its applicability through numerical simulation. [Methods] A self-developed model test apparatus was designed to simulate the penetration process of a cut-off wall into a clay core wall. Clay specimens were prepared to represent the core wall material, and three levels of overburden pressure (100, 300, and 500 kPa) were applied to investigate the influence of vertical stress conditions on the penetration behavior. A hyperbolic nonlinear normal contact model was proposed and calibrated to describe the mechanical relationship between wall-end pressure and penetration displacement. The proposed model was introduced into finite element numerical simulations to reproduce the penetration process to verify its effectiveness and applicability. [Results] The relationship between wall-end pressure and penetration displacement exhibits significant nonlinear characteristics during the penetration of the cut-off wall into the clay core wall. Under a constant overburden pressure, the wall-end resistance increases rapidly at the initial stage while the penetration displacement changes only slightly, indicating that the clay core wall provides a high initial resistance to wall penetration. As penetration continues, the increase rate of wall-end pressure gradually decreases, and the displacement begins to increase more noticeably. At the later stage, the penetration displacement increases substantially, whereas the growth rate of wall-end pressure becomes much smaller. This behavior indicates that the soil structure near the wall end is progressively disturbed and damaged, and large deformation of the clay core wall can occur without a proportional increase in contact pressure. The test results also demonstrate that overburden pressure has a significant influence on the penetration behavior. As the overburden pressure increases from 100 to 500 kPa, the overall bearing stiffness of the clay core wall increases markedly. This pressure-dependent behavior reflects the compaction effect and pressure-hardening characteristics of clay under higher vertical stress conditions. The proposed hyperbolic normal contact model effectively describes the measured relationship between wall-end pressure and penetration displacement. The coefficient of determination of the model fitting reaches 0.936 9, indicating good agreement between the fitted curves and the experimental data. The finite element results obtained using the proposed nonlinear normal contact model are generally consistent with the model test results, further confirming the rationality and effectiveness of the proposed contact model. [Conclusions] The penetration of a cut-off wall into a clay core wall is governed by a strongly nonlinear normal contact mechanism, rather than a simple linear pressure-displacement relationship. The wall-end resistance develops rapidly at the initial stage and then gradually approaches a slower growth trend as soil disturbance and local structural damage progress. Overburden pressure is a key controlling factor in this process, because higher vertical pressure enhances the compactness, stiffness, and resistance capacity of the clay core wall. The hyperbolic nonlinear normal contact model proposed in this study provides an effective mathematical representation of the wall-end pressure-penetration displacement relationship and can reasonably reflect the pressure-dependent deformation characteristics of clay. Its successful application in finite element analysis indicates that the model can serve as a reliable theoretical and numerical tool for analyzing the interaction between cut-off walls and clay core walls.