Med-ReLU: A Hybrid Activation Function Tailored for Deep Artificial Neural Networks in Medical Image Segmentation without Parameter Tuning

Background: Deep learning (DL) is derived from the domain of Artificial Neural Network (ANN). It makes one of the most important elements of deep learning algorithms. Deep learning segmentation models are based on layer-by-layer convolution learning attribute representation directed by forward and backward propagation. Throughout the process vital role is played by appropriately chosen activation function (AF) in order to guarantee the robustness of the model learning. However, the existing activation functions are either ineffective in addressing the vanishing gradient problem or get burdened with multiple parameters that need to be manually tuned. Moreover, the current research on activation function design mainly focuses on classification tasks using natural images from the MNIST, CIFAR-10 and CIFAR-100 datasets. Therefore,Med-ReLU as a novel activation function for medical image segmentation, is proposed. The proposed activation function avoids deep learning models from the attacks of dead neurons or from the vanishing gradient problems. Method: Med-ReLU is a hybrid activation function that combines the property of two activation functions of ReLU and Softsign. For positive inputs, Med-ReLU utilizes the linear property just like ReLU to produce an output without vanishing gradient. The negative inputs converge in polynomial ways towards their asymptotes as property of the softsign AF that ensures robust training processing without the problem of dead neurons that rarely activate across the entire training dataset. Results: The training performance and segmentation accuracy of Med-ReLU have been investigated. The proposed function has demonstrated stable training and does not suffer from over-fitting. Hence, Med-ReLU has consistently outperformed the existing state-of-art activation functions in medical image segmentation tasks. Conclusion: Med-ReLU has been designed as a parameter-free activation function for DL image segmentation tasks. This activation function is easy-to-implement on complex and deep learning models. The utility of this research lies in affirming the impact of Med-ReLU on different Artificial Neural Network architectures and for various kinds of anomaly addressing tasks.