Due to the increasing uncertainty of data for higher transmission rate, the Forward Error Correction (FEC) devices need to provide more powerful error correcting capability for optical communication systems. As compared with traditional hard RS decoders, the soft RS decoders can perform substantial coding gain but require much higher hardware complexity. In this paper, a decision-confined algorithm is proposed to enhance the error correcting performance with an area-efficient architecture. The novelty is that, instead of decoding numerous possible transmitted codewords and choosing the most likely one, only one candidate sequence will be decoded after confining the degree of error-locator polynomial Λ(x). For RS (255,239) codes, simulation results confirm that our approach provides 0.4 dB performance gain at 10-4 CER over the hard RS decoders. The experimental result reveals that our soft decoder can achieve 2.56 Gb/s throughput in standard CMOS 90 nm technology while having similar complexity as a hard decoder. It can fit well for 10-40 Gb/s with 16 RS decoders in optical fiber systems and 2.5 Gb/s GPON applications.