Notes - Machine Learning MT23, Covariance and correlation
Flashcards
For a random variable $\pmb x \in \mathbb R^D$, what is the covariance matrix $\text{cov}(\pmb x)$ and what is $(\text{cov}(\pmb x)) _ {ij}$?
and
\[(\text{cov}(\pmb x))_{ij} = \text{cov}(X_i, X_j)\]Covariance depends on the scale of variables. Can you define the correlation $\text{corr}(X, Y)$, which is normalised between $\pm 1$?
Can you define the density function $p(\pmb x)$ for a multivariate Gaussian with covariance $\pmb \Sigma$ and mean $\pmb \mu$?
Suppose $\pmb \theta$ represents the parameters of some distribution with density function $p$. Can you define the likelihood of observing $(x _ 1, \ldots, x _ n)$, i.e. the probability of observing the data with parameter $\theta$?
The multivariate Gaussian with mean $\pmb \mu$ and covariance matrix $\pmb \Sigma$ is defined as:
\[p(\pmb x) = \frac{1}{(2\pi)^{D/2} |\pmb \Sigma|^{1/2}\,}\exp\left(-\frac 1 2(\pmb x - \pmb \mu)^T\pmb \Sigma^{-1} (\pmb x - \pmb \mu)\right)\]
What qualitative relationship is there between the eigenvectors of $\pmb \Sigma$ and the variance?
Order the eigenvalues $\pmb v _ 1, \cdots, \pmb v _ n$ of $\pmb \Sigma$ according to the size of the eigenvalues $ \vert \lambda _ 1 \vert \ge \vert \lambda _ 2 \vert \ge \cdots \ge \vert \lambda _ n \vert $. Then these eigenvectors correspond to the directions of greatest variance in the distribution.