================================================================
AI FLUENCY — CHEAT SHEET
Evals & Benchmarking
================================================================

MMLU, HumanEval, LLM-as-judge, regression suites, and production evaluation practices.
6 terms

----------------------------------------------------------------

* Benchmark Contamination
  When a benchmark's questions and answers leak into a model's training data, inflating its score without real generalization.
  e.g. A model scoring suspiciously high because MMLU questions appeared in its training data.

* HumanEval
  A benchmark of 164 Python programming problems that tests whether a model's generated code passes automated unit tests.
  e.g. Checking if a model's generated function correctly reverses a string.

* LLM-as-Judge
  Using a capable LLM to score or compare other models' outputs on subjective criteria like helpfulness or coherence, in place of human review.
  e.g. Asking a strong model to pick the better of two draft replies.

* MMLU
  A benchmark that measures a model's breadth of knowledge across 57 academic subjects using multiple-choice questions.
  e.g. Comparing two models' general knowledge scores across law, medicine, and STEM.

* pass@k
  A code-generation metric measuring how often at least one of k sampled solutions passes all tests for a given problem.
  e.g. pass@10 checks whether any of 10 sampled attempts solves the problem.

* Regression Suite
  A curated set of test prompts and expected behaviors run automatically after every change to catch broken behavior before it reaches production.
  e.g. Re-running 200 saved prompts whenever a system prompt is edited.

----------------------------------------------------------------
Tip: paste this file into your favorite AI assistant and ask to be tutored on it.