Courses/Deep AI Engineering/Tokens, Attention and Training

Tokens Are Not Words

After this lesson you can explain what a tokenizer does, why the same text is a different number of tokens in another language or in code, and why a model that writes essays cannot count the letters in a word.