Dynamic connection pruning for densely connected convolutional neural networks

Densely connected convolutional neural networks dominate in a variety of downstream tasks due to their extraordinary performance. However, such networks typically require excessive computing resources, which hinders their deployment on mobile devices. In this paper, we propose a dynamic connection pruning algorithm, which is a cost-effective method to eliminate a large amount of redundancy in densely connected networks. First, we propose a Sample-Evaluation process to assess the contributions of connections. Specifically, sub-networks are sampled from the unpruned network in each epoch, while the parameters of the unpruned network are subsequently updated and the contributions of the connections are evaluated based on the performance of the sub-networks. Connections with low contribution will be pruned first. Then, we search for the distribution of pruning ratios by the Markov process. Finally, we prune the network based on the connection contribution and pruning ratios learned in the above two stages and obtain a lightweight network. The effectiveness of our method is verified on both high-level and low-level tasks. On the CIFAR-10 dataset, the top-1 accuracy barely drops (-0.03%) when FLOPs are reduced by 46.8%. In the super-resolution task, our model remarkably outperforms other lightweight networks in both visual and quantitative experiments. These results verify the effectiveness and generality of our proposed method.