What a tiny GPT actually does

A 198k-parameter transformer trained from scratch on Alice's Adventures in Wonderland and Through the Looking-Glass. Type a prompt: the forward pass runs here in your browser, and you can open every attention head that produces the next word.

198k parameters 2 layers · 4 heads d = 64 context 32 tokens vocab — val loss —

Loading weights…

Q, K, V

Every token is projected three ways: a query (“what am I looking for?”), a key (“what do I offer?”) and a value (“what do I pass on?”). The score is q·k / √d, softmaxed over the past.

Why the triangle

The mask sets the future to −∞ before the softmax, so a token can only attend to itself and what came before. That is what makes generation possible: the same weights work for every prefix.

Layer 1 vs layer 2

Head patterns change with depth. Early heads usually track nearby words and punctuation; the second layer mixes those signals into something closer to “what kind of word comes next”.

What it cannot do

198k parameters, 72k tokens of text, a 32-token memory. It has no world knowledge — it learned the rhythm of one author's sentences and little else. That is the point: you can see all of it.

What a tiny GPT actually does

Prompt

Attention

Next token

Generate