Part II

Deep Networks: Modern

Practices

162

This part of the book summarizes the state of modern deep learning as it is

used to solve practical applications.

Deep learning has a long history and many aspirations. Several proposed

approaches have yet to entirely bear fruit. Several ambitious goals have yet to be

realized. These less-developed branches of deep learning appear in the ﬁnal part of

the book.

This part focuses only on those approaches that are essentially working tech-

nologies that are already used heavily in industry.

Modern deep learning provides a powerful framework for supervised learning.

By adding more layers and more units within a layer, a deep network can represent

functions of increasing complexity. Most tasks that consist of mapping an input

vector to an output vector, and that are easy for a person to do rapidly, can be

accomplished via deep learning, given suﬃciently large models and suﬃciently

large datasets of labeled training examples. Other tasks, that cannot be described

as associating one vector to another, or that are diﬃcult enough that a person

would require time to think and reﬂect in order to accomplish the task, remain

beyond the scope of deep learning for now.

This part of the book describes the core parametric function approximation

technology that is behind nearly all modern practical applications of deep learning.

We begin by describing the feedforward deep network model that is used to

represent these functions. Next, we present advanced techniques for regularization

and optimization of such models. Scaling these models to large inputs such as high-

resolution images or long temporal sequences requires specialization. We introduce

the convolutional network for scaling to large images and the recurrent neural

network for processing temporal sequences. Finally, we present general guidelines

for the practical methodology involved in designing, building, and conﬁguring an

application involving deep learning and review some of its applications.

These chapters are the most important for a practitioner—someone who wants

to begin implementing and using deep learning algorithms to solve real-world

problems today.

163