================================================================
AI FLUENCY — CHEAT SHEET
Transformer Architecture
================================================================

Attention, positional encoding, mixture-of-experts, architectural trade-offs.
6 terms

----------------------------------------------------------------

* Attention
  A mechanism that lets a model weigh how relevant every other token is when processing a given token.
  e.g. In "The cat sat because it was tired", attention links "it" to "cat".

* Feed-Forward Network (FFN)
  The per-token fully-connected layer inside each transformer block that follows attention, adding non-linear processing capacity.
  e.g. Roughly two-thirds of a transformer's parameters typically live here.

* Layer Normalization
  A technique that rescales activations within a layer to stabilize and speed up training of deep networks.
  e.g. Applied before or after each attention/FFN sub-block.

* Mixture of Experts (MoE)
  An architecture where only a subset of specialized "expert" sub-networks activate per token, increasing capacity without a proportional compute cost.
  e.g. A model with 8 experts might route each token through only 2 of them.

* Positional Encoding
  Information added to token representations so the model knows word order, since attention alone is order-agnostic.
  e.g. RoPE (rotary positional encoding) is used in many modern LLMs.

* Self-Attention
  Attention applied within a single sequence, letting every token attend to every other token in the same input.
  e.g. Resolving pronoun references across a paragraph.

----------------------------------------------------------------
Tip: paste this file into your favorite AI assistant and ask to be tutored on it.