Pattern Recognition, 4th Edition

Pruning Techniques

These techniques start training a sufficiently large network, and then they remove, in a stepwise procedure, the free parameters that have little influence on the cost function. There are two major methodological directions:

Methods Based on Parameter Sensitivity Calculations

Let us take for example the technique suggested in [Lecu 90]. Using a Taylor series expansion, the variation imposed on the cost function by parameter perturbations iswhereand i,j runs over all the weights. The derivatives can be computed via the back-propagation ...