In this paper, a memory efficient Fine Grain Scalability (FGS) coefficient encoding method is proposed to reduce the external memory access requirement. In the H.264/AVC Scalable Video Extension, the FGS coefficients encoding is frame based. However, the frame based mechanism results in the difficulty of hardware implementation due to large internal memory requirements and external memory accesses. Therefore, a non-uniform memory size design which can achieve low external memory access is proposed to realize the macroblock based FGS coefficients encoding. Compared to previous work, our proposed method can save at least 38KB external memory accesses per frame in average.