Blogmark

Andrej Karpathy’s latest blog post (he doesn’t post very often, so you wanna check it out when he does) presents an implementation of a GPT in 200 lines of python with no dependencies.

This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency.