Single image deraining aims to reconstruct a clean rain-free background image from a single image deteriorated by rain streaks and rain accumulation. However, actual rain streaks have various densities, shapes, and directions, so it is not easy to decompose rainy image into a clean background layer and rain steaks layers. This paper proposes an U-Net based single image deraining network using the wavelet residue channel fusion strategy. The proposed residue channel is computed in wavelet low-frequency domain to learn enough background information, and combined with four wavelet subbands in fusion block, allowing them to take advantage of their respective strengths. To extract the main features for rain removal, we propose the extraction module based on LSTM and SE_Res. Furthermore, we present an encoder-decoder subnetwork based on U-Net structure to capture multiscale information of rain steaks. To evaluate the performance of the proposed network, experiments were performed on widely used benchmarks, and the results show that our method outperforms some popular single image deraining methods.