Lecture - Theories of Deep Learning MT25, XVI, Ingredients for a successful mini-project report

• Report should include:
  • A discussion of some theoretical portion of deep learning along
  • Numerical simulations
• Don’t write about an application (e.g. this method gets 2% better), this project is about the theory of deep learning
  • But you can highlight how an application raises questions about deep nets, and how the theory of deep learning could be used to overcome these problems
• One approach:
  • Pick two papers that came out at the same time and then compare them
  • Combine the two approaches
• You’re not expected to do conference-level research but there should be some aspect of originality. The examiners should hear your voice
  • E.g. if you’re comparing two papers, your report shouldn’t be useless if the examiner has read the two papers you have compared
• Pick a topic that you are excited about
  • It should read like a 20 page report that’s been compressed into a 5 page report
• Adapt your code from others
  • Pick papers that have code already to build on
• Don’t just focus on one paper
  • Compare different aspects of multiple papers
  • Look at papers that came out at the same time and haven’t been directly compared
• Work out what you don’t like when reading a paper, and don’t do that
• Clearly state what is new, be upfront and don’t present other results as your own
  • It can be jarring to say “I did X” but it makes the examiner’s life much easier
• Don’t pick papers that are very close to papers presented in the lectures
• You can pick older papers but don’t go back more than about 10 years

• You can copy figures from other papers but don’t make all your figures other people’s

• Examples
  • Robustness and accuracy: are we trying to have our cake and eat it too?
    • Nice bibliography
    • A nice mathematical tone
    • Novel experiment
    • Didn’t have a complete answer in the end
  • On manifold mixup for deep learning
    • Had a really good summary of the topic and the bibliography
    • Lots of originality, above what is expected
    • The examiner couldn’t find the new content anywhere else, it was genuinely new
  • Backpropagation and predictive coding: an experimental comparison
    • Contrasting
    • Ex
  • What were these papers missing to be exceptional?
    • Sometimes the lecturer gets
    • Great literature review
    • Really good experiments
    • With more page length it seems like it could be amazing
• Start with an outline
• Fill things in
• Go over the page length
• Then condense by selecting the most essential parts of the discussion

• Re-read and improve your report

• A literature review (done well) would get you a 60
• Putting some of yourself into the report would improve your score

• The more originality, the higher your score (roughly)

• Don’t generally recommend e.g. modifying a proof and doing lots of math, better to do numerical experiments instead