Transformer Architecture (3 blogmarks)

← Blogmarks

Transformers from Scratch, in Elixir

https://blog.jasondew.com/2026/09/09/transformers-from-scratch-in-elixir.html

So, I decided to implement a very small model in Elixir, completely from scratch – no libraries allowed. I cut the vocabulary down to just 32 words, used a single block and head of attention. It comes in at just over 15k parameters and trains in about 75 seconds on my laptop and gets the noun verb agreement correct.

LLM Transformer Architecture Elixir

Inventing Transformers

https://alexcbecker.net/blog/inventing-transformers.html

This post is a survey of several papers and techniques from 2012 to 2017 that were influential in the development of the modern Transformer Architecture. Which in turn is what is behind the rapid advancement in Large Language Models.

I like surveys like this because they give a high-level overview that can contextualize and point at both things I've already read about like Word2Vec as well as things that are new to me like Adam.

AI Transformer Architecture LLM Research

Transformer Explainer: LLM Transformer Model Visually Explained

https://poloclub.github.io/transformer-explainer/

Transformer is the core architecture behind modern Al, powering models like ChatGPT and Gemini. Introduced in 2017, it revolutionized how Al processes information. The same architecture is used for training on massive datasets and for inference to generate outputs. Here we use GPT-2 (small), simpler than newer ones but perfect for learning the fundamentals.

This explainer provides a nice set of text and visuals going through how the Transformer architecture works.

Here is the "Attention Is All You Need" paper that ushered in all these advances in large language models.

LLM Transformer Architecture