Uncertainty in Deep Learning MT25, Statistics reference
The notes here come primarily from the fourth chapter of Probabilistic Machine Learning by Kevin Murphy.
Flashcards
Maximum likelihood estimation
How can interpret the MLE in a Bayesian framework?
It is equivalent to a MAP estimate given a uniform prior over the parameters.
Kullback-Leibler divergence
@Define the Kullback-Leibler divergence $D _ {\mathbb {KL}}$ between two probability distributions $p$ and $q$.
\[\begin{aligned}
D _ {\mathbb{KL}}(p \parallel q) &= \sum _ {\pmb y} p(\pmb y) \log \frac{p(\pmb y)}{q(\pmb y)} \\
&= \sum _ {\pmb y} p(\pmb y) \log p(\pmb y) - \sum _ {\pmb y} p(\pmb y) \log q(\pmb y)
\end{aligned}\]