The Quality of Service (QoS) for applications is becoming increasingly important with the expansion of the Internet. A software-defined network offers flexibility and scalability in the use of network resources, like a meter table, which enables OpenFlow switches to measure flow rates and can be used to realize the rate limiter QoS. However, this drops packets when the flow-rate is over the limit, making the TCP congestion window reduce and grow periodically. The QoS of the upper layer using TCP will suffer regularly because of packet losses and bandwidth oscillation. Therefore, we propose a QoSTCP protocol using the meter and Rate Limiting Notification (RLN) to improve the QoS of TCP in SDN. The meter in the switch measures the flow rate to mark RLNs in the packets when the flow rate exceeds the threshold. A QoSTCP sender gets marking information to reduce the window size proportional to the marking packet ratio to avoid packet drop. We also provide a theoretical analysis to estimate when to mark packets at specific thresholds under two kinds of bandwidth demands in QoS, maximum and average rate. In order to verify QoSTCP performance, we implement a Two Rate Three Color Marker (trTCM) meter to measure the flow rate and mark RLNs using a Programming protocol-independent Packet Processors (P4) switch. To simplify our implementation, the existing Explicit Congestion Notification (ECN) and Data Center TCP (DCTCP) are modified to emulate QoSTCP behavior. Experiment results confirm not only that QoSTCP provides a consistent rate guarantee by avoiding packet drop but also that it matches our theoretical analytical results.