The fracture process of a heterogeneous brittle rock in the shear-box test is numerically investigated by means of the rock and tool interaction code (R-T2D). On the basis of the simulated results, the mode II fracture toughness is calculated, which is KIIC =7.72MPa root m, and the influence of heterogeneity and confinement on the formation and characteristics of shear fracture is discussed. It is found that with the confinement increasing, the fracture mechanisms in the shear-box test of heterogeneous brittle rock change from mixed tensile and shear failure but are dominated with tensile failure to dominant shear failure. It is concluded that the shear-box test under a confined condition is favourable for creating a condition for mode II fracture and a suitable method for measuring the mode II fracture toughness.