Unlabeled PCA-shuffling initialization for convolutional neural networks

In order to obtain prominent recognition accuracy convolutional neural networks (CNNs) need large amounts of labeled data to initialize network parameters. However, there exist two open problems, i.e., the uncertainties of the initialized effects and the limited labeled data To address the problems, we propose a novel method named UPSCNNs, which uses unlabeled data to perform Principal Component Analysis (PCA) and shuffling initialization for CNNs, composed of four steps, i.e. sampling the input images, calculating the sampling sets with PCA and initializing and shuffling the convolutional kernels. In cases with the same network architecture and activation function, i.e., Rectified Linear Units, we conduct the comparative experiments on three image datasets, i.e., STL-10, CIFAR-10(I) and CIFAR-10(II). In terms of accuracy, we find (1) the novel method increases by 4-20 percent in comparison to other weight initialization methods, e.g., Msra initialization, Xavier initialization and Random initialization and (2) an increase of 1-3 percent is obtained with unlabeled data than with only labeled data The results indicate that our method can make full use of unlabeled data for initializing CNNs to achieve good recognition effectiveness.