Python (2 blogmarks)

Andrej Karpathy’s latest blog post (he doesn’t post very often, so you wanna check it out when he does) presents an implementation of a GPT in 200 lines of python with no dependencies.

This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency.

These are three books recommended to read, in this order (but, of course, choose your own adventure):

• Effective Python
• Python Cookbook
• Fluent Python