Emerging applications require the location information of clients to enable human-environment interactions or personalized services. With an increasing number of antennas equipped in today’s wireless devices, recent research has shown the possibility of sub-meter level localization based only on the angle of arrival (AoA) of WiFi signals. While most existing work provides promising median accuracy, tail performance is usually far worse. We observe from measurements that the root cause is unequal AoA estimation reliability. In some critical areas, a small variation in the channel state information of signals could introduce an extremely large AoA estimation error. With this observation, we propose UAT (Unequal Angle Tracking), a confidence-aware AoA-based localization system. We show that unequal reliability of AoA measures can be mathematically quantified, allowing a system to weigh the estimates of different APs according to their confidence. Our testbed evaluation shows that UAT’s confidence-aware design provides reliable decimeter level localization for around 90% of locations. UAT is especially effective for unreliable areas and can reduce their localization errors by 27.5%, as compared to reliability-oblivious designs.