Acknowledgments
This book would not have been possible without the contributions of many people.
We would like to thank those who commented on our proposal for the book
and helped plan its contents and organization: Guillaume Alain, Kyunghyun Cho,
Çağlar Gülçehre, David Krueger, Hugo Larochelle, Razvan Pascanu and Thomas
Rohée.
We would like to thank the people who offered feedback on the content of the
book itself. Some offered feedback on many chapters: Martín Abadi, Ishaq Aden-Ali,
Guillaume Alain, Ion Androutsopoulos, Laura Ball, Fred Bertsch, Olexa Bilaniuk,
Ufuk Can Biçici, Matko Bošnjak, John Boersma, François Brault, Greg Brockman,
Alexandre de Brébisson, Pierre Luc Carrier, Sarath Chandar, Pawel Chilinski,
Mark Daoust, Oleg Dashevskii, Laurent Dinh, Stephan Dreseitl, Gudmundur
Einarsson, Hannes von Essen, Jim Fan, Miao Fan, Meire Fortunato, Frédéric
Francis, Nando de Freitas, Çağlar Gülçehre, Jurgen Van Gael, Yaroslav Ganin,
Javier Alonso García, Aydin Gerek, Stefan Heil, Jonathan Hunt, Gopi Jeyaram,
Chingiz Kabytayev, Lukasz Kaiser, Varun Kanade, Asifullah Khan, Akiel Khan,
John King, Diederik P. Kingma, Dominik Laupheimer, Yann LeCun, Minh Lê, Max
Marion, Rudolf Mathey, Matías Mattamala, Abhinav Maurya, Vincent Michalski,
Kevin Murphy, Oleg Mürk, Hung Ngo, Roman Novak, Augustus Q. Odena, Simon
Pavlik, Karl Pichotta, Eddie Pierce, Kari Pulli, Roussel Rahman, Tapani Raiko,
Anurag Ranjan, Johannes Roith, Mihaela Rosca, Halis Sak, César Salgado, Grigory
Sapunov, Yoshinori Sasaki, Mike Schuster, Julian Serban, Nir Shabat, Ken Shirriff,
Andre Simpelo, Scott Stanley, David Sussillo, Ilya Sutskever, Carles Gelada Sáez,
Graham Taylor, Valentin Tolmer, Massimiliano Tomassoli, An Tran, Shubhendu
Trivedi, Alexey Umnov, Vincent Vanhoucke, Robert Viragh, Marco Visentini-
Scarzanella, Martin Vita, David Warde-Farley, Dustin Webb, Shan-Conrad Wolf,
Kelvin Xu, Wei Xue, Ke Yang, Li Yao, Zygmunt Zając and Ozan Çağlayan.
We would also like to thank those who provided us with useful feedback on
individual chapters:
ix
CONTENTS
• Notation: Zhang Yuanhang.
•
Chapter 1, Introduction: Yusuf Akgul, Sebastien Bratieres, Samira Ebrahimi,
Charlie Gorichanaz, Benned Hedegaard, Brendan Loudermilk, Petros Mani-
atis, Eric Morris, Cosmin Pârvulescu, Muriel Rambeloarison, Alfredo Solano
and Timothy Whelan.
•
Chapter 2, Linear Algebra: Amjad Almahairi, Nikola Banić, Kevin Bennett,
Philippe Castonguay, Oscar Chang, Eric Fosler-Lussier, Andrey Khalyavin,
Sergey Oreshkov, István Petrás, Dennis Prangle, Thomas Rohée, Gitanjali
Gulve Sehgal, Colby Toland, Alessandro Vitale and Bob Welland.
•
Chapter 3, Probability and Information Theory: John Philip Anderson, Kai
Arulkumaran, Ana-Maria Cretu, Vincent Dumoulin, Rui Fa, Stephan Gouws,
Artem Oboturov, Patrick Pan, Antti Rasmus, Alexey Surkov and Volker
Tresp.
•
Chapter 4, Numerical Computation: Tran Lam An, Ian Fischer, William
Gandler, Mahendra Kariya and Hu Yuhuang.
•
Chapter 5, Machine Learning Basics: Dzmitry Bahdanau, Mark Cramer,
Eric Dolores, Justin Domingue, Ron Fedkiw, Nikhil Garg, Guillaume de
Laboulaye, Jon McKay, Makoto Otsuka, Bob Pepin, Philip Popien, Klaus
Radke, Emmanuel Rayner, Eric Sabo, Imran Saleh, Peter Shepard, Kee-Bong
Song, Zheng Sun, Alexandre Torres and Andy Wu.
•
Chapter 6, Deep Feedforward Networks: Uriel Berdugo, Fabrizio Bottarel,
Elizabeth Burl, Ishan Durugkar, Jeff Hlywa, Jong Wook Kim, David Krueger
and Aditya Kumar Praharaj.
•
Chapter 7, Regularization for Deep Learning: Brian Bartoldson, Morten
Kolbæk, Kshitij Lauria, Inkyu Lee, Sunil Mohan, Hai Phong Phan and
Joshua Salisbury.
•
Chapter 8, Optimization for Training Deep Models: Marcel Ackermann,
Tushar Agarwal, Peter Armitage, Rowel Atienza, Andrew Brock, Max Hayden
Chiz, Gregory Galperin, Aaron Golden, Russell Howes, Hill Ma, Tegan
Maharaj, James Martens, Kashif Rasul, Thomas Stanley, Klaus Strobl,
Nicholas Turner and David Zhang.
•
Chapter 9, Convolutional Networks: Martín Arjovsky, Eugene Brevdo, Jane
Bromley, Konstantin Divilov, Eric Jensen, Mehdi Mirza, Alex Paino, Guil-
laume Rochette, Marjorie Sayer, Ryan Stout and Wentao Wu.
x
CONTENTS
•
Chapter 10, Sequence Modeling: Recurrent and Recursive Nets: Gökçen
Eraslan, Nasos Evangelou, Steven Hickson, Christoph Kamann, Martin
Krasser, Razvan Pascanu, Diogo Pernes, Ryan Pilgrim, Lorenzo von Ritter,
Rui Rodrigues, Dmitriy Serdyuk, Dongyu Shi, Kaiyu Yang and Ruiqing Yin.
• Chapter 11, Practical Methodology: Daniel Beckstein and Kenji Kaneda.
•
Chapter 12, Applications: George Dahl, Vladimir Nekrasov and Ribana
Roscher.
• Chapter 13, Linear Factor Models: Jayanth Koushik.
• Chapter 14, Autoencoders: Hassan Masum.
•
Chapter 15, Representation Learning: Mateo Torres-Ruiz , Kunal Ghosh and
Rodney Melchers.
•
Chapter 16, Structured Probabilistic Models for Deep Learning: Deng Qingyu
, Harry Braviner, Timothy Cogan, Diego Marez, Anton Varfolom and Victor
Xie.
•
Chapter 18, Confronting the Partition Function: Sam Bowman and Jin Kim.
• Chapter 19, Approximate Inference: Yujia Bao.
•
Chapter 20, Deep Generative Models: Nicolas Chapados, Daniel Galvez,
Wenming Ma, Fady Medhat, Shakir Mohamed and Grégoire Montavon.
• Bibliography: Lukas Michelbacher, Leslie N. Smith and Max Xie.
We also want to thank those who allowed us to reproduce images, figures or
data from their publications. We indicate their contributions in the figure captions
throughout the text.
We would like to thank Lu Wang for writing pdf2htmlEX, which we used
to make the web version of the book, and for offering support to improve the
quality of the resulting HTML. We also thank Simon Lefrançois for incorporating
MIT Press’s edits to our manuscript back into the web edition, and for helping
incorporate reader feedback from the web.
We would like to thank Ian’s wife Daniela Flori Goodfellow for patiently
supporting Ian during the writing of the book as well as for help with proofreading.
We would like to thank the Google Brain team for providing an intellectual
environment where Ian could devote a tremendous amount of time to writing this
xi
CONTENTS
book and receive feedback and guidance from colleagues. We would especially like
to thank Ian’s former manager, Greg Corrado, and his current manager, Samy
Bengio, for their support of this project. Finally, we would like to thank Geoffrey
Hinton for encouragement when writing was difficult.
xii
