Course - Optimisation for Data Science HT25

Created: January 20, 2025 | Updated: January 26, 2026 | About these notes | View in context | Study these flashcards

---

This course analyses optimisation methods suitable for large-scale data science problems, mainly by deriving results on the rate of convergence under increasing assumptions (smooth, convex, strongly convex) on the objective functions.

The course begins with some optimisation terminology and then covers gradient descent and the proximal method, which can be used to apply steepest descent techniques to regularised problems. Then it covers acceleration techniques such as the heavy ball method, and then moves onto stochastic gradient descent and accelerated techniques in that context. Finally, it covers coordinate descent methods.

• Course Webpage
• Lecture Notes
• From the previous year:
  • Lecture Notes
  • 1, Scope and examples
  • 2, Terminology and prerequisites
  • 3, Method of steepest descent
  • 4, The proximal method
  • 5, Acceleration of gradient methods
  • 6, Stochastic gradient descent
  • 7, Reducing the noise floor in SGD
  • 8, Coordinate descent
  • 9, Practical coordinate descent
  • 10, Outlook (non-examinable)
• Other courses this term: [[Courses HT25]]^U

Notes

• [[Notes - Optimisation for Data Science HT25, Overview of convergence results]]^U ⭐️

• [[Notes - Optimisation for Data Science HT25, Motivation and examples]]^U
• [[Notes - Optimisation for Data Science HT25, Optimisation terminology]]^U
• [[Notes - Optimisation for Data Science HT25, Smoothness and convexity]]^U
• [[Notes - Optimisation for Data Science HT25, Subgradients]]^U
• [[Notes - Optimisation for Data Science HT25, Steepest descent]]^U
• [[Notes - Optimisation for Data Science HT25, Linesearch methods]]^U
• [[Notes - Optimisation for Data Science HT25, Proximal method]]^U
• [[Notes - Optimisation for Data Science HT25, Heavy ball method]]^U
• [[Notes - Optimisation for Data Science HT25, Nesterov’s accelerated gradient method]]^U
• [[Notes - Optimisation for Data Science HT25, Stochastic gradient descent]]^U
• [[Notes - Optimisation for Data Science HT25, Variance reduction methods]]^U

• [[Notes - Optimisation for Data Science HT25, Coordinate descent]]^U

• [[Notes - Optimisation for Data Science HT25, Useful miscellany]]^U
• [[Notes - Optimisation for Data Science HT25, Homemade exam questions]]^?

Problem Sheets

• [[Notes - Optimisation for Data Science HT25, Problem sheets]]^?
• Sheet 1, partial answers
• Sheet 2, partial answers
• Sheet 3, partial answers
• Sheet 4, partial answers

To-Do List

Relevant reading

• “Optimisation for Data Science” course at ETH Zurich
• “Convex Optimisation” at CMU
• Convex Optimisation, Boyd and Vandenberghe
• Numerical Optimisation, Nocedal and Wright
• Deep Learning, Goodfellow
• Optimisation for Machine Learning, Sra, Nowozin and Wright

---

Related posts

• [[Courses HT25]]^U

  (incoming)
• [[Part B]]^U

  (incoming)
• [[University Notes]]^U

  (incoming)
• [[Course - Continuous Optimisation HT26]]^U

  (incoming)
• [[Notes - Continuous Optimisation HT26, Overview of convergence results]]^U

  (sim: 0.696)
• [[Course - Theories of Deep Learning MT25]]^U

  (incoming)
• [[Lecture - Theories of Deep Learning MT25, VII, Stochastic gradient descent and its extensions]]^U

  (incoming)
• [[Lecture - Theories of Deep Learning MT25, VIII, Optimisation algorithms for training DNNs]]^U

  (incoming)
• [[Notes - Machine Learning MT23, Optimisation]]^U

  (incoming)
• [[Lecture - Machine Learning MT23, IX]]^U

  (sim: 0.69)
• [[Notes - Optimisation for Data Science HT25, Overview of convergence results]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Variance reduction methods]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Optimisation terminology]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Heavy ball method]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Subgradients]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Proximal method]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Linesearch methods]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Steepest descent]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Stochastic gradient descent]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Motivation and examples]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Smoothness and convexity]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Coordinate descent]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Useful miscellany]]^U

  (incoming)
• [[Notes - Optimisation for Data Science HT25, Nesterov's accelerated gradient method]]^U

  (incoming)