RLHF, Constitutional AI, jailbreaks, prompt injection, red teaming.
An alignment approach where a model critiques and revises its own outputs against a written set of principles, reducing reliance on human labeling for every case.
e.g. A model checking its own answer against a rule like "avoid harmful instructions."Automated checks (filters, classifiers, rules) placed around a model to catch or block unsafe or off-policy outputs.
e.g. A content filter blocking a response before it reaches the user.A prompt crafted to bypass a model's safety training and get it to produce disallowed content.
e.g. Disguising a harmful request as a "fictional story".An attack where malicious instructions are hidden in content a model processes (a webpage, document, email) to hijack its behavior.
e.g. A webpage containing hidden text telling an AI assistant to leak private data.Deliberately probing a model or system for weaknesses, harmful outputs, or exploitable behavior before it is deployed.
e.g. A security team trying dozens of jailbreak prompts before launch.A training technique that uses human preference judgments to reward a model for outputs people rate as better, aligning it with human intent.
e.g. Raters rank two model responses; the model is trained to prefer the higher-ranked one.