Adaptive Filtering

7	Newton’s and Steepest Descent Methods

7.1    ONE-DIMENSIONAL GRADIENT SEARCH METHOD

In general, we can say that adaptive algorithms are nothing but iterative search algorithms derived from minimizing a cost function with the true statistics replaced by their estimates. To study the adaptive algorithms, it is necessary to have a thorough understanding of the iterative algorithms and their convergence properties. In this chapter, we discuss the steepest descent and the Newton’s method.

The one-coefficient mean-square error (MSE) surface (line in this case) is given by [see (6.15)]

$J\left(w\right)=J_{\min }+r_{xx}\left(0\right){\left(w-w^o\right)}^2;J\left(w^o\right)\le J\left(w\right)\text{for}\text{all}w\prime \text{s}$

(7.1)

and it is pictorially shown in Figure 7.1. The first and second derivatives are

$\frac{\partial J\left(w\right)}{\partial w}=2r_{xx}($