Continuous Optimisation HT26, Taylor's theorem
Flashcards
Suppose:
- $f : \mathbb R^n \to \mathbb R$
- $f \in \mathcal C^1(\mathbb R^n)$
- $f$ has gradient $\nabla f = \left( \frac{\partial f}{\partial x _ 1}, \ldots, \frac{\partial f}{\partial x _ n} \right)$
- $x = (x _ 1, \ldots, x _ n)^\top$
- $s = (s _ 1, \ldots, s _ n)^\top$
@State the first order Taylor expansion of $f$ in this context.
for some $\tilde \alpha \in (0, \alpha)$.
Suppose:
- $f : \mathbb R^n \to \mathbb R$
- $f \in \mathcal C^2(\mathbb R^n)$
- $f$ has gradient $\nabla f = \left( \frac{\partial f}{\partial x _ 1}, \ldots, \frac{\partial f}{\partial x _ n} \right)$ and (symmetric) Hessian $(\nabla^2 f) _ {ij} = \frac{\partial^2 f}{\partial x _ i \partial x _ j}$
- $x = (x _ 1, \ldots, x _ n)^\top$
- $s = (s _ 1, \ldots, s _ n)^\top$
@State the second-order Taylor expansion of $f$ in this context.
for some $\tilde \alpha \in (0, \alpha)$.
Bite-sized
For $\phi : \mathbb R \to \mathbb R$ with $\phi \in \mathcal C^1(\mathbb R)$, the mean-value form of the first-order Taylor expansion is $\phi(\alpha) = \phi(0) + \alpha \phi'(\tilde \alpha)$ for some $\tilde \alpha \in (0, \alpha)$.
For $\phi : \mathbb R \to \mathbb R$ with $\phi \in \mathcal C^2(\mathbb R)$, the mean-value form of the second-order Taylor expansion is $\phi(\alpha) = \phi(0) + \alpha \phi'(0) + \tfrac{1}{2} \alpha^2 \phi''(\tilde \alpha)$ for some $\tilde \alpha \in (0, \alpha)$.
Define $\phi(\alpha) := f(x + \alpha s)$ for fixed $x, s \in \mathbb R^n$ and $f \in \mathcal C^1(\mathbb R^n)$. By the chain rule, $\phi'(\alpha) = \nabla f(x + \alpha s)^\top s$.
Define $\phi(\alpha) := f(x + \alpha s)$ for fixed $x, s \in \mathbb R^n$ and $f \in \mathcal C^2(\mathbb R^n)$. Then $\phi''(\alpha) = s^\top \nabla^2 f(x + \alpha s) \, s$.