Attention Mechanisms

Explorations into training LLMs to use clinical calculators from patient history, using open sourced models. Will start with Wells' Criteria

Implementation of fused cosine similarity attention in the same style as Flash Attention

LocoFormer - Generalist Locomotion via Long-Context Adaptation

Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible experimentation and ex…

Explorations into the recently proposed Taylor Series Linear Attention

Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with Sparse Transformers"