In this paper, a flexible two-layer external memory management for H.264/AVC decoder is proposed. Power consumption and data access latency caused by being fetched to/from the off-chip memory greatly affect multimedia system performance. The proposed memory controller consists of two layers. The first layer is the address translation which provides an efficient pixel data arrangement to reduce the row-miss occurrence. The second layer is the external memory interface (EMI) which can further reduce access latency up to 70% by using the specific command FIFO and a unified FSM with generic scheduling. Particularly, the memory utilization can be increased about 3 times as compared with traditional method after combining the address translation layer with external memory interface. Similarly, the proposed memory controller unit is feasible and beneficial for future memory-bandwidth-constraint System-on-Chip applications.