Blogmark

From the author of Build a Large Language Model (from scratch), here is a jupyter notebook that implements a basic BPE (byte-pair encoding) tokenizer for use with an LLM. Training, prompting, and responses make use of a fixed BPE tokenizers. In practice you probably want to make use of an optimized implementation like tiktoken from OpenAI. That said, it can still be informative and fun to build your own from scratch.

When I started working through the Build an LLM book, I wanted to get a better understanding of BPE, so I wrote my own tokenizer in Python.