Polymer-dispersed liquid crystals (PDLCs) modulate the amplitude and optical phase of light. The optical phase modulation of PDLC can be dissected into two parts: Kerr phase and orientational phase according to the electro-optical (EO) response. We investigated the origins of the Kerr and orientational phases in PDLCs and their connection with the two-step EO response. The Kerr phase is attributed to LC orientation in the center of LC droplets. The orientational phase results from orientation of LC molecules near LC–polymer interfaces. Both phases can be adjusted by varying the droplet size. The two-step EO response in small droplets (<333 nm) is related to the Kerr and orientational phases, and possibly to rotation of point defects. A modified PDLC model considering the Kerr and orientational phases is proposed. Our findings suggest the possibility of versatile photonic devices using pure optical phase modulation.