In line with the basic principle of First Order Second Moment method, a nonlinear function involving many influencing factors, including the height of retaining wall, the internal friction angle of filling material, the unit weight soil behind the retaining wall, the coefficient of friction between soil and wall base, the friction angle between soil and wall back, is established to calculate the reliability of balance-weight retaining wall, hence judging the stability accurately. Two failure modes, sliding failure and overturn failure, are considered. The nonlinearity of physical and mechanical properties of geomaterials and the possible second fracture surface are also taken into account. Results suggest that the present function could solve the problem caused by nonlinear earth pressure on the back of retaining wall by calculating the reliability of anti-sliding stability and anti-overturn stability in consideration of the possible second fracture surface. Compared with linear model, the present nonlinear model is worth promotion in engineering as the calculated reliability is smaller, hence is safer and more suitable for practice.