self attention cv

The implementations cover attention mechanisms that operate on 2D spatial inputs rather than the 1D sequences of original NLP transformers, addressing the computational challenges of applying attention across image pixelswhere naive all-pairs attention is quadratic in image size.

Implementations include efficient variants using factored attention along spatial axes, local window attention, and relative position encoding approaches that maintain the spatial inductive biases useful for image understanding tasks while achieving the long-range dependency modeling that attention enables.

Computer vision researchers studying hybrid CNN-attention architectures, engineers implementing vision transformer variants for specific applications, and practitioners looking for reference implementations before the technique was absorbed into mainstream frameworks use this repository.

The clean PyTorch implementations serve as pedagogical references for understanding how self-attention adapts from its NLP origins to 2D spatial data, and as starting points for researchers developing novel attention mechanisms tailored to specific vision challenges like video understanding or dense prediction tasks.