Lecture - Theories of Deep Learning MT25, VII, Stochastic gradient descent and its extensions

Created: November 14, 2025 | Updated: November 15, 2025 | About these notes

---

• [[Course - Theories of Deep Learning MT25]]^U

This lecture and the next ([[Lecture - Theories of Deep Learning MT25, VIII, Optimisation algorithms for training DNNs]]^U) are effectively a mini-speedrun of [[Course - Optimisation for Data Science HT25]]^U. In particular, this lecture covered results on the convergence of stochastic gradient descent and how to decrease the noise floor:

• [[Notes - Optimisation for Data Science HT25, Stochastic gradient descent]]^U
• [[Notes - Optimisation for Data Science HT25, Variance reduction methods]]^U

---

Related posts

• [[Course - Theories of Deep Learning MT25]]^U

  (outgoing)
• [[Lecture - Theories of Deep Learning MT25, VIII, Optimisation algorithms for training DNNs]]^U

  (outgoing)
• [[Lecture - Theories of Deep Learning MT25, XIII, Autoencoders]]^U

  (sim: 0.7)
• [[Lecture - Theories of Deep Learning MT25, V, Controlling the exponential growth of variance and correlation]]^U

  (sim: 0.691)
• [[Course - Optimisation for Data Science HT25]]^U

  (outgoing)
• [[Notes - Optimisation for Data Science HT25, Stochastic gradient descent]]^U

  (outgoing)
• [[Notes - Optimisation for Data Science HT25, Variance reduction methods]]^U

  (outgoing)
• [[Lecture - Machine Learning MT23, XIV]]^U

  (sim: 0.685)
• [[Lecture - Machine Learning MT23, XVI]]^U

  (sim: 0.667)