Differential Energy Watermarking (DEW) based techniques have been applied in DEW, modified schemes of DEW or Wavelet Tree Quantization (WTQ) algorithms by differentiating the transformed coefficient energy to verify the watermark embedding for copyright protection and ownership verification. In this paper, we present a novel differential energy watermarking algorithm based on the wavelet tree group modulation (WTGM) structure. The wavelet coefficients of the image are divided into disjoint tree groups and each tree group contains two sub tree groups. The watermark is embedded in the tree components using the group strategy such that the energy of tree groups is close and each sub-group performs different modulation for watermark embedding. Therefore, the employment of wavelet tree structure, sum-of-subsets and positive/negative modulation effectively improve the watermark robustness. In addition, the contrast sensitive function (CSF) and noise visibility function (NVF) of human visual system are also considered for the better visual quality of the watermarked image. The experimental results demonstrate the effectiveness of WTGM algorithm in terms of robustness and imperceptibility of watermarking.