Latte: Latent Attention for Linear Time Transformers

This repository is an implementation of latent attention (latte) for language modelling.

Get started

Get started by cloning the repository.

We use PDM to manage Python packages and dependencies and use Python 3.11 (consider pyenv as a simple Python version management solution). pdm is a modern alternative to pip / poetry. pdm manages a virtualenv locally in the project directory itself (in a directory .venv).

It keeps dependencies from pyproject.toml in sync with the virtual environment. This makes it easy to have a python environment specific to the project and ensure we all run the same dependencies. Similar to how npm, yarn for JS/TS or cargo for Rust work.

Installation

Once you have pdm and the right python version installed, run

pdm install

Scripts

• pdm run train will train a model
• pdm run preprocess-tiny-stories will download and preprocess the TinyStories dataset locally