This paper investigates new decentralized dynamic resource allocation scheme which exploits game theoretic regret-matching procedure to enhance the spectrum efficiency for cognitive femtocell networks. In order to ensure spectrum usage, we develop a new resource allocation scheme with joint overlay and underlay strategy to increase frequency reuse factor among cognitive femtocell networks. The proposed scheme can avoid interference to macrocell and the other femtocell user equipments; furthermore, the number of iterations required is less than the original regret-matching scheme, and it has been proved that at least one of correlated equilibrium exists in proposed algorithm. Computer simulations are presented to verify the proposed scheme which is effective for resource allocation. The proposed scheme is comparable to the performance of exhaustive search scheme and outperforms the original regret-matching scheme.