LLM Mechanics

How LLMs run: sampling, context windows, embeddings, KV cache, quantization.

The maximum number of tokens (input plus output) a model can consider at one time.

e.g. A 200K-token context window can hold roughly a 500-page book.

A dense numeric vector representing the meaning of text, used for search and similarity comparisons.

e.g. "dog" and "puppy" have embeddings that sit close together.

Cached attention key/value tensors from prior tokens that let a model avoid recomputing them for every new token, speeding up generation.

e.g. Long conversations reuse the cache instead of reprocessing all prior history.

The raw, unnormalized scores a model outputs for each possible next token before they are converted into probabilities.

e.g. A softmax function turns logits into a probability distribution.

Reducing the numeric precision of a model's weights (e.g. 16-bit to 4-bit) to shrink memory use and speed up inference, at some accuracy cost.

e.g. Running a large model on a laptop GPU via 4-bit quantization.

Controls used to pick the next token from the model's probability distribution; temperature adjusts how random that choice is.

e.g. Temperature 0 gives deterministic, repeatable answers.