All Research
SafetyRLHFConstitutional AIAlignment

Constitutional AI: Harmlessness from AI Feedback

We introduce a method for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The core idea is to use a list of rules about harmlessness—a constitution—to automatically correct outputs. We train both a supervised policy and a reinforcement learned policy, and find that the RL policy achieves better results than RLHF while generating fewer harmful outputs.

Published
Authors
Dr. Sarah ChenMarcus WebbDr. Aisha PatelJames LiuDr. Elena Vasquez

Overview

Constitutional AI represents a fundamental shift in how we approach AI safety training. Rather than relying solely on human labelers to identify harmful content, we leverage the model’s own capabilities to generate, critique, and revise its outputs according to a predefined set of principles.

Key Contributions

  1. Self-Critique Pipeline: Models can identify and correct their own harmful outputs using natural language feedback
  2. Constitutional Principles: A set of 16 principles derived from human rights documents, safety research, and common sense ethics
  3. RLAIF: Reinforcement Learning from AI Feedback, reducing human annotation costs by 97%

Results

Our Constitutional AI approach achieves state-of-the-art harmlessness scores while maintaining competitive helpfulness ratings. The model trained via RLAIF demonstrates superior calibration compared to RLHF baselines.