CADRE

CADRE ("Catholic Alignment, Doctrine, and Reasoning Evaluation") evaluates how well AI language models understand and articulate Catholic teaching. The benchmark tests models across the four pillars of the Catechism, emphasizing dogmatic teachings that form the foundation of Catholic faith.

Our methodology incorporates the hierarchy of truths, weighting questions by authoritative level: Dogma (Level 1), Definitive Doctrine (Level 2), Authentic Magisterium (Level 3), and Prudential Judgments (Level 4).

CADRE uses two question variants to distinguish between factual retrieval and native reasoning patterns. Explicit questions test whether models can accurately retrieve Catholic teaching when directly asked. Implicit questions test whether Catholic reasoning emerges naturally when presented with neutrally-phrased questions answerable from multiple theological perspectives. This dual approach reveals whether a model merely possesses Catholic knowledge versus whether it exhibits Catholic reasoning as its native mode of ethical and theological analysis.

The benchmark consists of 50 questions across the four pillars of the Catechism—Creed (15 questions), Sacraments (12), Moral Life (13), and Prayer (10)—weighted by the hierarchy of truths: 56% dogma (divinely revealed truths), 26% definitive doctrine (magisterial teaching), and 18% authentic magisterium (authoritative teaching).

Each question has two variants testing distinct capabilities. Explicit questions assess whether models can retrieve Catholic teaching when directly asked, testing precise doctrinal knowledge, theological terminology, and citation ability (e.g., "What is the Catholic Church's teaching on the Holy Trinity?"). Implicit questions evaluate whether Catholic reasoning emerges naturally without prompting, using lenient scoring on details but evaluating native alignment patterns (e.g., "What is the relationship between the Father, Son, and Holy Spirit?"). This dual approach reveals whether a model merely has Catholic knowledge in its training data versus whether it exhibits Catholic reasoning as its default worldview.

LLM-as-judge (Claude Opus 4.1) evaluates responses using structured rubrics with weighted criteria: 3-5 criteria per question with assigned weights, required versus optional criteria (failures on required criteria result in zero scores), reference answers with magisterial sources (CCC, councils, encyclicals), and assessment of theological precision, factual accuracy, and absence of error.