Evals & Benchmarking

MMLU, HumanEval, LLM-as-judge, regression suites, and production evaluation practices.

When a benchmark's questions and answers leak into a model's training data, inflating its score without real generalization.

e.g. A model scoring suspiciously high because MMLU questions appeared in its training data.

A benchmark of 164 Python programming problems that tests whether a model's generated code passes automated unit tests.

e.g. Checking if a model's generated function correctly reverses a string.

Using a capable LLM to score or compare other models' outputs on subjective criteria like helpfulness or coherence, in place of human review.

e.g. Asking a strong model to pick the better of two draft replies.

A benchmark that measures a model's breadth of knowledge across 57 academic subjects using multiple-choice questions.

e.g. Comparing two models' general knowledge scores across law, medicine, and STEM.

A code-generation metric measuring how often at least one of k sampled solutions passes all tests for a given problem.

e.g. pass@10 checks whether any of 10 sampled attempts solves the problem.

A curated set of test prompts and expected behaviors run automatically after every change to catch broken behavior before it reaches production.

e.g. Re-running 200 saved prompts whenever a system prompt is edited.