Best Practice Guide - Deep Learning

Artificial neural networks (ANNs) are a class of machine learning models that are loosely inspired by the biological neural networks that constitute animal brains. Artificial neural networks use multiple layers of nonlinear processing units for feature extraction and transformation. This allows models to represent multiple levels of abstraction from the data: an approach that works well for many types of problems, such as image, sound, and text analysis. Neural networks with many layers are known as deep neural networks, and the process of training such networks is known as ‘deep learning’. Sparked by various inference and prediction challenges on publicly available large datasets (such as MS-COCO [1], ImageNet, Open Images, Yelp Reviews, the Wikipedia Corpus, WMT, Merk Molecular Activity, Million Songs and FMA) and by having available open-source frameworks (such as TensorFlow, Caffe and PyTorch), the deep learning field has evolved rapidly over the past decade. With more complex neural networks and larger input data sets, the scalability of deep learning algorithms is an increasingly important topic. The main aim of this guide is to teach you how to perform deep learning at large scale. In the process, different algorithms, software frameworks and hardware platforms will be discussed. This should help you pick the most suitable framework and hardware platform for your deep learning problem. However, note that this guide only gives a broad overview and does not aim to replace software framework manuals or a deep learning course. The guide is structured in five chapters: Hardware, Algorithms, HPC and scaling, Frameworks and Use cases. The separation into these different chapters is occasionally difficult, as the concepts are closely related: the type of framework you choose may be dependent upon the hardware you want to run on, which algorithms it supports, etc. We recommend reading the full guide if you want to get a complete picture.