Tokens, Attention and Training
Enough of the machinery to explain a failure: how text becomes tokens, what attention does with them, what the training stages put in and leave out, and the mechanical reasons behind the mistakes the earlier courses taught you to catch.
- Lessons
- 6
- Exercises
- 34
- Minutes
- 40
- 1
Tokens Are Not Words
After this lesson you can explain what a tokenizer does, why the same text is a different number of tokens in another language or in code, and why a model that writes essays cannot count the letters in a word.
DebugMultiple choiceTrace - 2
Embeddings and Attention
After this lesson you can say what an embedding is, what attention computes at each token, why that makes context the whole game, and why long contexts cost what they cost.
Fill blankMultiple choiceTrace - 3
What Training Puts In
After this lesson you can name the training stages, say what each one changes, and explain why a model knows so much, knows it fuzzily, and behaves as it does.
DebugMultiple choiceTapTrace - 4
Every Failure Has a Mechanism
After this lesson you can take a failure from any earlier course and name the mechanism behind it, which is what tells you whether a prompt, a design change or nothing at all will fix it.
DebugFill blankMultiple choice - 5
Checkpoint: The MachineryCheckpoint
Tokens, attention, training and mechanisms in fresh situations.
Fill blankMultiple choiceTrace - 6
Boss: Explain the FailureBoss
One week of incidents at Ledgerly, an invoicing product with three AI features. Eight failures, each solved by naming the machinery behind it and choosing the lever that actually moves it.
DebugCode orderFill blankMultiple choice