Notes - Continuous Optimisation HT26, Taylor's theorem

Suppose:

$f : \mathbb R^n \to \mathbb R$
$f \in \mathcal C^1(\mathbb R^n)$
$f$ has gradient $\nabla f = \left( \frac{\partial f}{\partial x _ 1}, \ldots, \frac{\partial f}{\partial x _ n} \right)$
$x = (x _ 1, \ldots, x _ n)^\top$
$s = (s _ 1, \ldots, s _ n)^\top$

@State the first order Taylor expansion of $f$ in this context.

\[f(x + \alpha s) = f(x) + \alpha \nabla f(x + \tilde \alpha s)^\top s\]

for some $\tilde \alpha \in (0, \alpha)$.

Suppose:

$f : \mathbb R^n \to \mathbb R$
$f \in \mathcal C^2(\mathbb R^n)$
$f$ has gradient $\nabla f = \left( \frac{\partial f}{\partial x _ 1}, \ldots, \frac{\partial f}{\partial x _ n} \right)$ and (symmetric) Hessian $(\nabla^2 f) _ {ij} = \frac{\partial^2 f}{\partial x _ i \partial x _ j}$
$x = (x _ 1, \ldots, x _ n)^\top$
$s = (s _ 1, \ldots, s _ n)^\top$

@State the second-order Taylor expansion of $f$ in this context.

\[f(x + \alpha s) = f(x) + \alpha \nabla f(x)^\top s + \frac 1 2 \alpha^2 s^\top \nabla^2 f(x + \tilde \alpha s)s\]