We quantitatively investigate the influence of high-order transverse modes on the self-mode locking (SML) in an optically pumped semiconductor laser (OPSL) with a nearly hemispherical cavity. A physical aperture is inserted into the cavity to manipulate the excitation of high-order transverse modes. Experimental measurements reveal that the laser is operated in a well-behaved SML state with the existence of the TEM0,0 mode and the first high-order transverse mode. While more high-order transverse modes are excited, it is found that the pulse train is modulated by more beating frequencies of transverse modes. The temporal behavior becomes the random dynamics when too many high-order transverse modes are excited. We observe that the temporal trace exhibits an intermittent mode-locked state in the absence of high-order transverse modes.