How to Jump-Start Your Deep Learning Research
Florian Knoll1

1New York University School of Medicine, New York, NY, United States


This talk will provide a practical hands-on overview of how to get started in machine learning research from the point of view of an imaging lab. Common hurdles and pitfalls will be discussed via didactic examples from classification and reconstruction. The key differences of medical imaging data and computer vision applications will be highlighted. The talk will also discuss software frameworks and implementation, including code demos, which will be made available as open source.


How to get started in machine learning research from the point of view of an imaging lab.

Overview of educational talk

Recent developments in deep learning1 have led to breakthrough improvements in areas as diverse as image classication2 semantic labelling3, optical flow4, image restauration5 or playing the game of Go6 . Recently, attempts have been made to leverage neural networks for medical image reconstruction7,8,9,10,11,12,13. The goal of this educational talk is to help bridging the gap between having read research papers on deep learning, and applying these techniques to dedicated research projects in an imaging lab. Overlaps of medical imaging with general pattern recognition and computer vision will be discussed as well as the unique elements encountered in medical imaging. The talk will be organized around didactic examples, using well known datasets from pattern recognition14, computer vision15 and custom radiology data. Implementation of fully connected and convolutional neural networks will be discussed in general purpose computing environments like Matlab (The MathWorks, Natick, MA, USA) as well as specialized deep learning libraries like Tensorflow16 and high level interfaces like Keras17. Source code of all presented examples will be made available online. Particular areas of emphasis that will be discussed in the talk are:

  • Collection and organization of data.
  • Selection of an appropriate network architecture.
  • Selection of an implementation framework.
  • Selection and optimization of network hyper-parameters.
  • Training a model: Selecting a training algorithm, hyper-parameters and the loss function, monitoring of training performance.
  • Evaluation of a trained model: Training, validation and test error.
  • Over- and underfitting. Evaluation of robustness, generalization and transfer to related problems.


While deep learning and neural networks open up exciting possibilities in Radiology, translation of developments from computer vision and pattern recognition are sometimes not straight-forward. The development of approaches that are both robust and practical enough so that they can replace currently used clinical methods is still an open research topic.


NIH P41 EB017183, NVIDIA corporation.


[1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, 436–444 (2015).

[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 1097–1105 (2012).

[3] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs,” in International Conference on Learning Representations (2015).

[4] A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbas ̧, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox, “FlowNet: Learning Optical Flow with Convolutional Networks,” in IEEE International Conference on Computer Vision (ICCV), 2758–2766 (2015).

[5] Y. Chen, W. Yu, and T. Pock, “On learning optimized reaction diffusion processes for effective image restoration,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 5261–5269 (2015).

[6] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489 (2016).

[7] K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, and F. Knoll, “Learning a Variational Network for Reconstruction of Accelerated MRI Data,” Magn. Reson. Med., in press (2017).

[8] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, D. Liang, “Accelerating magnetic resonance imaging via deep learning”, ISBI 514-517 (2016). [9] K.H. Jin, M.T. McCann, E. Froustey, M, Unser, “Deep Convolutional Neural Network for Inverse Problems in Imaging”, (2016).

[10] K. Kwon, D. Kim, H. Seo, J. Cho, B. Kim, H.W. Park, “Learning-based Reconstruction using Artificial Neural Network for Higher Acceleration”, in Proceedings of the International Society of Magnetic Resonance in Medicine (ISMRM), p1801 (2016).

[11] G. Wang, “Perspective on Deep Imaging”, IEEE Access 8914-8924 (2016)

[12] V Golkov, A Dosovitskiy, J.I. Sperl, M.I. Menzel, M. Czisch, P. Saemann, T. Brox, D. Cremers, “q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI Scans”, IEEE TMI 35: 1344-1351 (2016).

[13] F Knoll, K Hammernik, E Garwood, A Hirschmann, L Rybak, M Bruno, T Block, J Babb, T Pock, DK Sodickson and MP Recht, “Accelerated knee imaging using a deep learning based reconstruction” in Proceedings of the International Society of Magnetic Resonance in Medicine (ISMRM) p645 (2017).

[14] R. A. Fisher. "The use of multiple measurements in taxonomic problems". Annals of Eugenics 7: 179–188 (1936).

[15] LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. “Gradient-based learning applied to document recognition”. Proceedings of the IEEE, 86, 2278–2324 (1998).

[16] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, X. Zheng, and G. Brain, “TensorFlow: A System for Large-Scale Machine Learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16), 265–284 (2016).

[17] François Chollet, “Keras” GitHub (2015).

Proc. Intl. Soc. Mag. Reson. Med. 26 (2018)