Notes - Machine Learning MT23, Matrix calculus
Flashcards
Write $\mathbf c^T \mathbf w$ as a sum.
\[\sum^D_{j = 0} c_j w_j\]
Write $\mathbf w^T \mathbf A \mathbf w$ as a sum.
\[\sum^D_{i = 0} \sum^D_{j = 0} w_iw_j A_{ij}\]
Write $\mathbf{AB}$ as a sum, giving the expression for $c _ {ij}$.
\[c_{ij} = \sum^N_{k=1} a_{ik} b_{kj}\]
Suppose:
- $\pmb z \in \mathbb R^n$
- $f : \mathbb R^n \to \mathbb R$
Can you define
\[\frac{\partial f}{\partial \pmb z}\]
?
\[\frac{\partial f}{\partial \pmb z} = \left[ \frac{\partial f}{\partial z_1}, \dots, \frac{\partial f}{\partial z_n} \right]\]
Suppose:
- $\pmb z \in \mathbb R^n$
- $\pmb f : \mathbb R^n \to \mathbb R^m$, $(\pmb f(\pmb z)) _ i = f _ i(\pmb z)$
Can you define
\[J_{\pmb z}(\pmb f ) = \frac{\partial \pmb f}{\partial \pmb z}\]
?
\[\frac{\partial \pmb f}{\partial \pmb z} =
\begin{bmatrix}
\frac{\partial f_1}{\partial z_1} & \frac{\partial f_1}{\partial z_2} & \cdots & \frac{\partial f_1}{\partial z_n} \\\\
\frac{\partial f_2}{\partial z_1} & \frac{\partial f_2}{\partial z_2} & \cdots & \frac{\partial f_2}{\partial z_n} \\\\
\vdots & \vdots & \ddots & \vdots \\\\
\frac{\partial f_m}{\partial z_1} & \frac{\partial f_m}{\partial z_2} & \cdots & \frac{\partial f_m}{\partial z_n}
\end{bmatrix}\]
Suppose:
- $\pmb z \in \mathbb R^n$
- $f : \mathbb R^n \to \mathbb R$
- $\pmb W \in \mathbb R^{n \times m}$
Can you define
\[\frac{\partial f}{\partial \pmb W}\]
?
\[\frac{\partial f}{\partial \pmb W} =
\begin{bmatrix}
\frac{\partial f}{\partial W_{11}\\,} & \frac{\partial f}{\partial W_{12}\\,} & \cdots & \frac{\partial f}{\partial W_{1m}\\,} \\\\
\frac{\partial f}{\partial W_{21}\\,} & \frac{\partial f}{\partial W_{22}\\,} & \cdots & \frac{\partial f}{\partial W_{2m}\\,} \\\\
\vdots & \vdots & \ddots & \vdots \\\\
\frac{\partial f}{\partial W_{n1}\\,} & \frac{\partial f}{\partial W_{n2}\\,} & \cdots & \frac{\partial f}{\partial W_{nm}\\,}
\end{bmatrix}\]