Paper - Dynamics of Transient Structure in In-Context Linear Regression Transformers

• Full title: Dynamics of Transient Structure in In-Context Linear Regression Transformers
• Author(s): Liam Carroll, Jesse Hoogland, Matthew Farrugia-Roberts, Daniel Murfet
• Year: 2025
• Link: http://arxiv.org/abs/2501.17745
• Relevant for:
  • [[Course - Theories of Deep Learning MT25]]^U
• Links to:
  • [[Paper - In-Context Learning Strategies Emerge Rationally (2025)]]^?
• note by default this is public

Summary

• in-context linear regression
  • given some task vector $t \in \mathbb R^D$, sample a fixed length distribution of pairs $(x _ i, y _ i)$ where $x _ i$ is normally distributed and $y _ i \sim \mathcal N(y _ i \mid t^\top x _ i, \sigma^2)$
  • vary the “task diversity” by picking different

Flashcards