Paper - Dynamics of Transient Structure in In-Context Linear Regression Transformers
- Full title: Dynamics of Transient Structure in In-Context Linear Regression Transformers
- Author(s): Liam Carroll, Jesse Hoogland, Matthew Farrugia-Roberts, Daniel Murfet
- Year: 2025
- Link: http://arxiv.org/abs/2501.17745
- Relevant for:
- Links to:
Summary
- in-context linear regression
- given some task vector $t \in \mathbb R^D$, sample a fixed length distribution of pairs $(x _ i, y _ i)$ where $x _ i$ is normally distributed and $y _ i \sim \mathcal N(y _ i \mid t^\top x _ i, \sigma^2)$
- vary the “task diversity” by picking different