Gabriel Chartrand, PhD, Phillip M Cheng, MD, MS Eugene Vorontsov, BASc Eng Sci, Michal Drozdzal, PhD Simon Turcotte, MD, MSc, Christopher J Pal, PhD Samuel Kadoury, PhD, An Tang, MD, MSc

Author’s Summary

Deep learning is a class of machine learning methods that are gaining success and attracting interest in many domains, including computer vision, speech recognition, natural
language processing, and playing games.

Deep learning methods produce a mapping from raw inputs to desired outputs (eg, image classes). Unlike traditional machine learning methods, which require hand-engineered feature extraction from inputs, deep learning methods learn these features directly from data. With the advent of large datasets and increased computing power, these methods can produce models with exceptional performance. These models are multilayer artificial neural networks, loosely inspired by biologic neural systems.

Weighted connections between nodes (neurons) in the network are iteratively adjusted based on example pairs of inputs and target outputs by back-propagating a corrective error signal through the network. For computer vision tasks, convolutional neural networks (CNNs) have proven to be effective. Recently, several clinical applications of CNNs have been proposed and studied in radiology for classification, detection, and segmentation tasks. This article reviews the key concepts of deep learning for clinical radiologists, discusses technical requirements, describes emerging applications in clinical radiology, and outlines limitations and future directions in this field. Radiologists should become familiar with the principles and potential applications of deep learning in medical imaging.


  • Deep learning is a type of representation learning in which the algorithm learns a composition of features that reflect a hierarchy of structures in the data. Complex representations are expressed in terms of simpler representations.
  • Although neural networks have been used for decades, in recent years three key factors have enabled the training of large neural networks: (a) the availability of large quantities of labelled data, (b) inexpensive and powerful parallel computing hardware,
    and (c) improvements in training techniques and architectures.
  • The basic unit of an artificial neural network, the artificial neuron or node, is a simplified model that mimics the basic mechanism found in the biologic neuron.
  • Deep CNNs exploit the compositional structure of natural images so that shifts and deformations of objects in the images do not significantly affect the overall performance of the network.
  • The creation of these large databases of labeled medical images and many associated challenges will be fundamental to foster future research in deep learning applied to medical images.

A view from
Anthony Chang, MD, MBA, MPH, MS

“This article is absolutely the best single review article on the topic of deep learning for image intensive clinicians, particularly radiologists. The subject of deep learning is reviewed in reasonable detail and the concepts of deep learning are well covered. Not only are the text explanations excellent, but the figures are also outstanding so that with this coupling, difficult concepts such as softmax classifier, learning algorithm, and convolution are very easily understood even for the data science novice. In addition, the clinical application section is strong so that clinicians can relate to this in the clinical setting. Overall, it is difficult to find another article that can better enlighten anyone on the use of data science and deep learning in imaging.”