AI UX PlaygroundNewsletterJoin 2K+ AI designers and PMs on Substack. New teardowns, patterns, and prompts as they drop.

AI Workflows

Build a scoring rubric for AI output quality

Turn “is the AI output good?” into a weighted 1–5 rubric with anchor examples, failure-type mapping, and a calibration set so different reviewers score the same way.

Prompt by

Marie Claire DeanOwl-Listener

PhD in Conversational AI; led UX at Google (Maps, Shopping, Labs) and built design teams at Atlassian

MIT License

GitHub

Prompt

You are building a quality evaluation rubric.
Follow this process:
## Step 1: Define What You're Evaluating
- What AI feature or output type is this rubric for?
- What does the user expect from this output?
- What are the highest-priority quality attributes?
## Step 2: Select Quality Dimensions
Using **output-quality-rubrics**:
- Select relevant quality dimensions from: accuracy, relevance, completeness, helpfulness, clarity, tone appropriateness, safety
- Add any domain-specific dimensions needed
- Define each dimension in concrete terms for this specific output type
## Step 3: Build the Scoring Scale
Using **output-quality-rubrics**:
- For each dimension, create a 1-5 scale with detailed descriptions at each level
- Write anchor examples at levels 1, 3, and 5
- Define what differentiates each adjacent level
## Step 4: Weight the Dimensions
Using **output-quality-rubrics** and **task-success-metrics**:
- Assign weights to each dimension based on importance for the user's task
- Justify the weighting
- Define the overall score calculation
## Step 5: Map to Failure Types
Using **failure-taxonomy**:
- For each low score (1-2) on each dimension, identify the corresponding failure type
- This connects the rubric to the failure tracking system
## Step 6: Design Calibration
Using **output-quality-rubrics**:
- Create a calibration set of 5 outputs with pre-scored ratings
- Write evaluator instructions
- Define the calibration process for new evaluators
## Output
Deliver a complete evaluation rubric:
1. Dimension definitions with scoring scales (1-5)
2. Anchor examples at levels 1, 3, and 5 for each dimension
3. Dimension weights and overall score formula
4. Failure type mapping
5. Calibration set with pre-scored outputs
6. Evaluator instructions

How to use

  1. 01

    Above the prompt, describe the AI feature and paste 3–5 real outputs from it, good and bad.

  2. 02

    Paste the prompt into Claude or ChatGPT.

  3. 03

    Check the anchor examples at levels 1, 3, and 5 against your own judgment, and edit any you disagree with.

  4. 04

    Have two teammates score the calibration set separately, then compare scores to see where the rubric is unclear.

Tips

  • Real outputs make the anchor examples concrete. Without them, the model invents examples.
  • The bold names (output-quality-rubrics, task-success-metrics, failure-taxonomy) are skills in MC Dean’s AI Design Skills plugin. The steps work without it.
  • Removed one line from the original: "Use only skills from the evaluation plugin." It only applies inside the plugin.