Evaluating LLM output quality: metrics, benchmarks, and automated grading for Claude agents

By the end of this tutorial, you’ll have a working Python framework that automatically grades Claude agent outputs against baselines using BLEU, ROUGE, semantic similarity, and LLM-as-judge scoring — with results you can track over time. If you’re deploying Claude in production and flying blind on output quality, this fixes that.

Most teams can’t consistently evaluate LLM output quality until something breaks in production. A customer complains, a hallucinated fact slips through, or a regression sneaks in after a prompt change. The framework below gives you deterministic metrics plus heuristic scoring in a single pipeline — so you catch degradation before users do.

1. Install dependencies — set up the evaluation environment with required libraries
2. Define your test dataset — build a reproducible baseline of prompt/expected-output pairs
3. Run lexical metrics (BLEU, ROUGE) — score n-gram overlap against reference outputs
4. Add semantic similarity scoring — catch correct answers that use different wording
5. Build an LLM-as-judge grader — use Claude to score Claude on rubric dimensions
6. Aggregate scores and log results — persist everything for trend tracking

Why Lexical Metrics Alone Will Fail You

BLEU and ROUGE were designed for machine translation and summarisation, where there’s a well-defined reference text. They measure n-gram overlap — great for checking if a summary includes key phrases, useless for open-ended generation tasks. A Claude agent answering “What’s the capital of France?” with “Paris is the capital of France” scores worse against a reference of “The capital is Paris” than a wrong answer that happens to share more tokens.

The honest take: use lexical metrics as a sanity check and regression detector, not as your primary quality signal. They’re cheap, deterministic, and reproducible — which is exactly what you want for CI/CD integration. Semantic similarity and LLM-as-judge fill the gaps for nuanced quality assessment.

If you’re already thinking about how hallucinations interact with quality scoring, our article on reducing LLM hallucinations in production covers verification patterns that pair well with the grading framework here.

Step 1: Install Dependencies

# Core evaluation libraries
pip install anthropic nltk rouge-score sentence-transformers numpy pandas

# NLTK data for BLEU scoring
python -c "import nltk; nltk.download('punkt'); nltk.download('punkt_tab')"

Versions that work at time of writing: anthropic>=0.25, sentence-transformers>=2.7, rouge-score>=0.1.2. Pin these — sentence-transformers in particular has had breaking API changes.

Step 2: Define Your Test Dataset

Your evaluation is only as good as your test cases. Build a dataset of prompt/expected-output pairs that represent real production traffic — not synthetic examples you made up in five minutes.

import json

# eval_dataset.json structure — keep this in version control
EVAL_DATASET = [
    {
        "id": "summarisation_01",
        "prompt": "Summarise the following in 2 sentences: Claude is an AI assistant made by Anthropic...",
        "reference": "Claude is an AI assistant developed by Anthropic, designed to be helpful, harmless, and honest.",
        "task_type": "summarisation",
        "rubric": {
            "accuracy": "Does the summary accurately reflect the source?",
            "conciseness": "Is it within the requested length?",
            "tone": "Is the tone neutral and professional?"
        }
    },
    {
        "id": "extraction_01",
        "prompt": "Extract the company name, date, and total amount from this invoice text: ...",
        "reference": '{"company": "Acme Corp", "date": "2024-01-15", "total": "$1,250.00"}',
        "task_type": "extraction",
        "rubric": {
            "accuracy": "Are all three fields correctly extracted?",
            "format": "Is the output valid JSON?"
        }
    }
]

# Save locally for repeatability
with open("eval_dataset.json", "w") as f:
    json.dump(EVAL_DATASET, f, indent=2)

Aim for 20–50 test cases per task type in production. Fewer gives you noisy results; more becomes expensive if you’re running LLM-as-judge scoring on each eval run.

Step 3: Run Lexical Metrics (BLEU, ROUGE)

from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
from rouge_score import rouge_scorer
from typing import Dict

def compute_lexical_metrics(hypothesis: str, reference: str) -> Dict[str, float]:
    """
    Compute BLEU-4 and ROUGE-L scores for a single hypothesis/reference pair.
    Both inputs should be plain text strings.
    """
    # BLEU: tokenise and smooth to handle short outputs
    ref_tokens = reference.lower().split()
    hyp_tokens = hypothesis.lower().split()
    
    smoother = SmoothingFunction().method1  # avoids zero scores on short texts
    bleu = sentence_bleu(
        [ref_tokens], 
        hyp_tokens, 
        weights=(0.25, 0.25, 0.25, 0.25),  # BLEU-4
        smoothing_function=smoother
    )
    
    # ROUGE-L: longest common subsequence — better for summaries
    scorer = rouge_scorer.RougeScorer(['rougeL', 'rouge1'], use_stemmer=True)
    scores = scorer.score(reference, hypothesis)
    
    return {
        "bleu_4": round(bleu, 4),
        "rouge_l_f1": round(scores['rougeL'].fmeasure, 4),
        "rouge_1_f1": round(scores['rouge1'].fmeasure, 4)
    }

# Quick test
result = compute_lexical_metrics(
    "Paris is the capital of France.",
    "The capital of France is Paris."
)
print(result)  # {'bleu_4': 0.3015, 'rouge_l_f1': 0.6154, 'rouge_1_f1': 0.6667}

Step 4: Add Semantic Similarity Scoring

This is where you catch semantically correct answers that score poorly on lexical overlap. The all-MiniLM-L6-v2 model runs locally, costs nothing per call, and produces embeddings in under 100ms on CPU.

from sentence_transformers import SentenceTransformer
import numpy as np

# Load once at module level — this model is ~80MB
_embed_model = SentenceTransformer('all-MiniLM-L6-v2')

def semantic_similarity(text_a: str, text_b: str) -> float:
    """
    Returns cosine similarity [0, 1] between two texts.
    Scores above 0.85 generally indicate semantic equivalence.
    """
    embeddings = _embed_model.encode([text_a, text_b], normalize_embeddings=True)
    similarity = float(np.dot(embeddings[0], embeddings[1]))
    return round(similarity, 4)

# Test: semantically equivalent, lexically different
score = semantic_similarity(
    "The quarterly revenue increased by 12 percent.",
    "Revenue grew 12% in Q3 compared to the previous period."
)
print(score)  # ~0.87 — correctly identified as semantically close

For domain-specific tasks (legal, medical, finance), consider fine-tuning a bi-encoder or using a domain-specific model. all-MiniLM-L6-v2 is good enough for general tasks but will miss technical equivalences. This connects closely to the concepts in our semantic search implementation guide.

Step 5: Build an LLM-as-Judge Grader

This is the highest-signal metric for nuanced tasks — and the most expensive. A Claude call costs roughly $0.003 per evaluation using claude-3-5-haiku-20241022 (about $0.0008 input + $0.002 output at current pricing for a short grading prompt). Run this on every eval in CI and you’re looking at ~$0.15 per full dataset pass at 50 cases — very manageable.

import anthropic
import json

client = anthropic.Anthropic()  # uses ANTHROPIC_API_KEY from env

JUDGE_SYSTEM_PROMPT = """You are an impartial evaluator assessing AI assistant responses.
Score the response on each rubric dimension from 1-5.
Return ONLY valid JSON with dimension names as keys and integer scores as values.
Include a brief "reasoning" key explaining your scores in one sentence."""

def llm_judge_score(
    prompt: str, 
    response: str, 
    rubric: dict
) -> dict:
    """
    Uses Claude to score a response against a rubric.
    Returns dict of dimension -> score (1-5) plus reasoning.
    """
    rubric_text = "\n".join([f"- {k}: {v}" for k, v in rubric.items()])
    
    user_message = f"""Original prompt: {prompt}

AI response to evaluate:
{response}

Scoring rubric (score each 1-5, where 5 is excellent):
{rubric_text}

Return JSON only."""

    message = client.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=256,
        system=JUDGE_SYSTEM_PROMPT,
        messages=[{"role": "user", "content": user_message}]
    )
    
    try:
        scores = json.loads(message.content[0].text)
    except json.JSONDecodeError:
        # Occasionally the model adds markdown fences — strip them
        raw = message.content[0].text.strip().strip("```json").strip("```")
        scores = json.loads(raw)
    
    return scores

One practical note: LLM-as-judge scores have positional bias and can be inconsistent for borderline cases. Run each evaluation twice with shuffled rubric order and average the results if you need high confidence — or accept ±0.5 variance as noise on single-run scoring.

Step 6: Aggregate Scores and Log Results

import anthropic
import pandas as pd
import datetime
import json
from pathlib import Path

def run_evaluation_suite(dataset: list, log_path: str = "eval_results.jsonl") -> pd.DataFrame:
    """
    Runs full evaluation pipeline on a dataset.
    Logs results to JSONL for trend tracking. Appends, doesn't overwrite.
    """
    client = anthropic.Anthropic()
    results = []
    run_timestamp = datetime.datetime.utcnow().isoformat()
    
    for case in dataset:
        # 1. Get Claude's actual response
        message = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=512,
            messages=[{"role": "user", "content": case["prompt"]}]
        )
        hypothesis = message.content[0].text.strip()
        
        # 2. Lexical metrics
        lexical = compute_lexical_metrics(hypothesis, case["reference"])
        
        # 3. Semantic similarity
        sem_sim = semantic_similarity(hypothesis, case["reference"])
        
        # 4. LLM judge (skip if no rubric defined)
        judge_scores = {}
        if case.get("rubric"):
            judge_scores = llm_judge_score(case["prompt"], hypothesis, case["rubric"])
        
        # 5. Composite score — weighted average (tune weights for your domain)
        judge_avg = (
            sum(v for k, v in judge_scores.items() if k != "reasoning" and isinstance(v, (int, float))) /
            max(1, len([k for k in judge_scores if k != "reasoning" and isinstance(judge_scores[k], (int, float))]))
        ) / 5.0  # normalise to [0, 1]
        
        composite = (
            0.2 * lexical["rouge_l_f1"] +
            0.3 * sem_sim +
            0.5 * judge_avg
        ) if judge_scores else (0.4 * lexical["rouge_l_f1"] + 0.6 * sem_sim)
        
        record = {
            "run_ts": run_timestamp,
            "case_id": case["id"],
            "task_type": case["task_type"],
            "hypothesis": hypothesis,
            **lexical,
            "semantic_similarity": sem_sim,
            "judge_scores": judge_scores,
            "composite_score": round(composite, 4)
        }
        results.append(record)
        
        # Append to JSONL log
        with open(log_path, "a") as f:
            f.write(json.dumps(record) + "\n")
    
    df = pd.DataFrame(results)
    print(df[["case_id", "rouge_l_f1", "semantic_similarity", "composite_score"]].to_string())
    return df

# Run it
df = run_evaluation_suite(EVAL_DATASET)
print(f"\nMean composite score: {df['composite_score'].mean():.3f}")

The JSONL append pattern means you can track score trends over time across model versions and prompt changes. Load multiple runs into pandas and plot the composite score history to spot regressions immediately after prompt edits.

For teams running high-volume agent systems, this evaluation loop integrates naturally with observability platforms — we compared Helicone, LangSmith, and Langfuse in detail if you want to pipe these scores into a managed dashboard rather than managing JSONL files yourself.

Common Errors

JSONDecodeError from the judge model

Claude occasionally wraps JSON in markdown fences despite being told not to, especially on short rubrics. The fix is already in the code above — strip ```json and ``` before parsing. If it’s still failing, log the raw response and check: the model may be returning an explanation instead of JSON when it finds the task ambiguous. Tighten the system prompt with an explicit “Return ONLY the JSON object, no other text” instruction.

BLEU scores of 0.0 for short outputs

BLEU-4 requires at least 4 tokens in both hypothesis and reference to score above zero. For outputs under ~15 words, use BLEU-1 or BLEU-2 instead, or rely entirely on ROUGE-L and semantic similarity. The SmoothingFunction().method1 helps but doesn’t fully solve sub-4-gram cases.

Sentence-transformers model loading on every call

If you’re initialising SentenceTransformer inside your evaluation function, you’re reloading 80MB into memory on every single call. Keep the model as a module-level singleton (as shown in the code above). In a FastAPI evaluation endpoint, initialise it in the application lifespan context.

Evaluating Prompting Strategy Changes

Once your baseline scores are established, the framework becomes a regression test suite for prompt engineering. Before and after adding few-shot examples, changing the system prompt structure, or switching model versions — run the full eval suite and diff the composite scores. A drop of more than 0.05 in composite score on your core task types should be treated as a regression.

This pairs directly with systematic prompting experiments. If you’re iterating on prompting strategies, our comparison of zero-shot vs few-shot prompting for Claude agents benchmarks the quality differences in a way that maps cleanly to the metrics framework here.

When to Use Which Metric

• BLEU/ROUGE: Regression detection, CI/CD gates, summarisation and translation tasks
• Semantic similarity: Open-ended Q&A, paraphrase-heavy tasks, any task where wording varies legitimately
• LLM-as-judge: Nuanced rubrics, tone assessment, instruction-following, anything where “correct” is multidimensional
• Composite score: Overall health dashboards, cross-version comparisons, go/no-go decisions on prompt changes

What to Build Next

Extend this framework into an A/B testing harness for prompts. Store each test run with a prompt_version field, run the same dataset against two system prompt variants in parallel, and use a paired t-test on composite scores to determine if the difference is statistically significant before shipping a prompt change. This turns your evaluation from a point-in-time quality check into a proper experimentation infrastructure — the kind of thing that separates teams shipping production AI reliably from teams guessing. If you also want to handle the failure modes that affect scores, the patterns in our guide on LLM fallback and retry logic are directly applicable to making your eval suite itself resilient.

Bottom line by reader type: Solo founders should start with just semantic similarity + LLM-as-judge on 15 representative test cases — you’ll get 80% of the signal for 20% of the setup effort. Teams with a prompt engineer or ML engineer should implement the full pipeline with JSONL logging and wire it into CI immediately. Enterprise teams should pipe composite scores into their existing observability stack and set automated alerts for score regressions across agent versions.