Field Guide
Inside the Forward Pass
Type a sentence and watch a language model read it — the way it actually does, one token at a time. The tokenizer below is the real GPT-2, running entirely in your browser.
STEP 01
Tokens Real GPT-2 · GPT-3
A model never sees words. It sees tokens — chunks of bytes from a fixed vocabulary of 50,257. Common words are one token; rare ones shatter into pieces. The leading space is part of the token (that is why ·is differs from is). Type anything:
This is the real byte-pair encoding GPT-2 and GPT-3 both used — one vocabulary, checked token for token against OpenAI’s own implementation. It loads once and runs locally — no server, nothing sent anywhere. Try emoji, code, numbers, or another language and watch it fragment.
STEP 02
Embeddings Real GPT-2
Each token id is swapped for a learned vector of 768 numbers — its meaning, written as coordinates. Here are the first 24 dimensions of your first few tokens.
These are GPT-2’s own values: the first 24 of the 768 numbers it learned for each token, teal where positive and amber where negative, kept at 4-bit precision — enough for the colour, not for arithmetic.
The six tokens whose vectors point most nearly the same way, measured across all 768 dimensions. Distance from the centre is real — one minus the cosine similarity — and the angle around it means nothing. Click one to move there.
STEP 03
Attention Schematic
Every token looks back at the tokens before it and decides how much each one matters. The grid is read row by row: row i shows how token i distributes its attention over tokens 1…i.
The weights are illustrative — but the lower-triangular shape is exactly right: a token can attend to the past, never the future. That single rule is what makes generation work left to right.
What happens after attention — position signals, the MLP, twelve stacked layers, and the projection onto 50,257 candidate next-tokens — is walked through in full in the essay this guide belongs to: How Machines Think →