Inference and Quantization
What happens when a model runs: how the next token is chosen, why the first token is slow and the rest are fast, what limits throughput, and what quantization buys and costs, for the engineer sizing a deployment.
- Lessons
- 7
- Exercises
- 40
- Minutes
- 45
- 1
Choosing the Next Token
After this lesson you can explain what the model actually outputs, how temperature and top-p turn it into one token, why temperature zero is not a guarantee, and what a stop token is.
Fill blankMultiple choiceTrace - 2
Prefill, Decode and the Cache
After this lesson you can explain why the first token is slow and the rest are fast, what the KV cache stores and why it fills memory, and why prefix caching works.
DebugFill blankMultiple choice - 3
Throughput Versus Latency
After this lesson you can explain why batching makes serving cheaper and each request slower, what continuous batching changes, and which number to optimise for which product.
Fill blankMultiple choiceTrace - 4
Quantization and Smaller Models
After this lesson you can say what quantization changes, estimate the memory it saves, name the quality risks, and choose between a quantized big model and a small one.
DebugFill blankMultiple choiceTap - 5
Serving in Production
After this lesson you can follow a request through a model server, read its metrics, and set the limits that make overload fail fast instead of slowly.
DebugCode orderMultiple choiceTrace - 6
Checkpoint: Running the ModelCheckpoint
Decoding, caches, batching and bits in fresh situations.
Fill blankMultiple choice - 7
Boss: Size the DeploymentBoss
One deployment for 200 engineers, from model choice to the first capacity incident: bits, memory, cache, batch, the number the product feels, and the measurement that decides it.
DebugFill blankMultiple choiceTrace