Acknowledgments
This book would not have been possible without the contributions of many people.
We would like to thank those who commented on our proposal for the book
and helped plan its contents and organization: Guillaume Alain, Kyunghyun Cho,
Çağlar Gülçehre, David Krueger, Hugo Larochelle, Razvan Pascanu and Thomas
Rohée.
We would like to thank the people who offered feedback on the content of the
book itself. Some offered feedback on many chapters: Martín Abadi, Guillaume
Alain, Ion Androutsopoulos, Fred Bertsch, Olexa Bilaniuk, Ufuk Can Biçici, Matko
Bošnjak, John Boersma, Greg Brockman, Alexandre de Brébisson, Pierre Luc
Carrier, Sarath Chandar, Pawel Chilinski, Mark Daoust, Oleg Dashevskii, Laurent
Dinh, Stephan Dreseitl, Jim Fan, Miao Fan, Meire Fortunato, Frédéric Francis,
Nando de Freitas, Çağlar Gülçehre, Jurgen Van Gael, Yaroslav Ganin, Javier
Alonso García, Jonathan Hunt, Gopi Jeyaram, Chingiz Kabytayev, Lukasz Kaiser,
Varun Kanade, Asifullah Khan, Akiel Khan, John King, Diederik P. Kingma, Yann
LeCun, Rudolf Mathey, Matías Mattamala, Abhinav Maurya, Kevin Murphy, Oleg
Mürk, Roman Novak, Augustus Q. Odena, Simon Pavlik, Karl Pichotta, Eddie
Pierce, Kari Pulli, Roussel Rahman, Tapani Raiko, Anurag Ranjan, Johannes
Roith, Mihaela Rosca, Halis Sak, César Salgado, Grigory Sapunov, Yoshinori
Sasaki, Mike Schuster, Julian Serban, Nir Shabat, Ken Shirriff, Andre Simpelo,
Scott Stanley, David Sussillo, Ilya Sutskever, Carles Gelada Sáez, Graham Taylor,
Valentin Tolmer, Massimiliano Tomassoli, An Tran, Shubhendu Trivedi, Alexey
Umnov, Vincent Vanhoucke, Marco Visentini-Scarzanella, Martin Vita, David
Warde-Farley, Dustin Webb, Kelvin Xu, Wei Xue, Ke Yang, Li Yao, Zygmunt
Zając and Ozan Çağlayan.
We would also like to thank those who provided us with useful feedback on
individual chapters:
Notation: Zhang Yuanhang.
ix
CONTENTS
Chapter 1, Introduction: Yusuf Akgul, Sebastien Bratieres, Samira Ebrahimi,
Charlie Gorichanaz, Brendan Loudermilk, Eric Morris, Cosmin Pârvulescu
and Alfredo Solano.
Chapter 2, Linear Algebra: Amjad Almahairi, Nikola Banić, Kevin Bennett,
Philippe Castonguay, Oscar Chang, Eric Fosler-Lussier, Andrey Khalyavin,
Sergey Oreshkov, István Petrás, Dennis Prangle, Thomas Rohée, Gitanjali
Gulve Sehgal, Colby Toland, Alessandro Vitale and Bob Welland.
Chapter 3, Probability and Information Theory: John Philip Anderson, Kai
Arulkumaran, Vincent Dumoulin, Rui Fa, Stephan Gouws, Artem Oboturov,
Antti Rasmus, Alexey Surkov and Volker Tresp.
Chapter 4, Numerical Computation: Tran Lam AnIan Fischer and Hu
Yuhuang.
Chapter 5, Machine Learning Basics: Dzmitry Bahdanau, Justin Domingue,
Nikhil Garg, Makoto Otsuka, Bob Pepin, Philip Popien, Emmanuel Rayner,
Peter Shepard, Kee-Bong Song, Zheng Sun and Andy Wu.
Chapter 6, Deep Feedforward Networks: Uriel Berdugo, Fabrizio Bottarel,
Elizabeth Burl, Ishan Durugkar, Jeff Hlywa, Jong Wook Kim, David Krueger
and Aditya Kumar Praharaj.
Chapter 7, Regularization for Deep Learning: Morten Kolbæk, Kshitij Lauria,
Inkyu Lee, Sunil Mohan, Hai Phong Phan and Joshua Salisbury.
Chapter 8, Optimization for Training Deep Models: Marcel Ackermann, Peter
Armitage, Rowel Atienza, Andrew Brock, Tegan Maharaj, James Martens,
Kashif Rasul, Klaus Strobl and Nicholas Turner.
Chapter 9, Convolutional Networks: Martín Arjovsky, Eugene Brevdo, Kon-
stantin Divilov, Eric Jensen, Mehdi Mirza, Alex Paino, Marjorie Sayer, Ryan
Stout and Wentao Wu.
Chapter 10, Sequence Modeling: Recurrent and Recursive Nets: Gökçen
Eraslan, Steven Hickson, Razvan Pascanu, Lorenzo von Ritter, Rui Rodrigues,
Dmitriy Serdyuk, Dongyu Shi and Kaiyu Yang.
Chapter 11, Practical Methodology: Daniel Beckstein.
Chapter 12, Applications: George Dahl, Vladimir Nekrasov and Ribana
Roscher.
x
CONTENTS
Chapter 13, Linear Factor Models: Jayanth Koushik.
Chapter 15, Representation Learning: Kunal Ghosh.
Chapter 16, Structured Probabilistic Models for Deep Learning: Minh
and Anton Varfolom.
Chapter 18, Confronting the Partition Function: Sam Bowman.
Chapter 19, Approximate Inference: Yujia Bao.
Chapter 20, Deep Generative Models: Nicolas Chapados, Daniel Galvez,
Wenming Ma, Fady Medhat, Shakir Mohamed and Grégoire Montavon.
Bibliography: Lukas Michelbacher and Leslie N. Smith.
We also want to thank those who allowed us to reproduce images, figures or
data from their publications. We indicate their contributions in the figure captions
throughout the text.
We would like to thank Lu Wang for writing pdf2htmlEX, which we used to
make the web version of the book, and for offering support to improve the quality
of the resulting HTML.
We would like to thank Ian’s wife Daniela Flori Goodfellow for patiently
supporting Ian during the writing of the book as well as for help with proofreading.
We would like to thank the Google Brain team for providing an intellectual
environment where Ian could devote a tremendous amount of time to writing this
book and receive feedback and guidance from colleagues. We would especially like
to thank Ian’s former manager, Greg Corrado, and his current manager, Samy
Bengio, for their support of this project. Finally, we would like to thank Geoffrey
Hinton for encouragement when writing was difficult.
xi