Mathematics of Deep Learning

9 Backpropagation

The training of deep neural networks (DNNs) requires to efficiently compute $\nabla \mathit{L}\mathrm{(}\mathit{\alpha }\mathrm{)}$, where the vector of parameters $\mathit{\alpha }=\mathrm{(}{\mathit{\alpha }}_{\mathit{k}}\mathrm{)}\in {\mathbb{R}}^{\mathit{\mu }}$ has large dimension $\mathit{\mu }\gg 1$; currently for the largest DNNs, μ can be more than half a trillion! Calculating individually all partial derivatives for each one of these μ parameters is time consuming, even for high-performance computers. Therefore, techniques for expediting the calculation of derivatives are valuable tools for training DNNs.

One of the most effective of these techniques is backpropagation, which takes advantage of the fact that $\mathit{L}\mathrm{(}\mathit{\alpha }\mathrm{)}$ is not an arbitrary function of many parameters. On the contrary, the dependence of $\mathit{L}\mathrm{(}\mathit{\alpha }\mathrm{)}$ on the parameters α is determined by the compositional structure of ...