Transformer Architecture

Attention, positional encoding, mixture-of-experts, architectural trade-offs.

A mechanism that lets a model weigh how relevant every other token is when processing a given token.

e.g. In "The cat sat because it was tired", attention links "it" to "cat".

The per-token fully-connected layer inside each transformer block that follows attention, adding non-linear processing capacity.

e.g. Roughly two-thirds of a transformer's parameters typically live here.

A technique that rescales activations within a layer to stabilize and speed up training of deep networks.

e.g. Applied before or after each attention/FFN sub-block.

An architecture where only a subset of specialized "expert" sub-networks activate per token, increasing capacity without a proportional compute cost.

e.g. A model with 8 experts might route each token through only 2 of them.

Information added to token representations so the model knows word order, since attention alone is order-agnostic.

e.g. RoPE (rotary positional encoding) is used in many modern LLMs.

Attention applied within a single sequence, letting every token attend to every other token in the same input.

e.g. Resolving pronoun references across a paragraph.