// EVAL CATALOG
Every eval, scorer and judge. All 47 of them.
Everything RunAgain can score a run with, from model-free string checks to calibrated LLM judges. Attach any of them online to live traffic or offline in experiments; parameters live in the docs.
Heuristic scorers
Deterministic, model-free, no API key required. Each compares an output to a reference (or checks the output alone) and returns 0 to 1. docs →
1 if the output equals the expected value exactly.
Normalized edit-distance similarity.
Numeric closeness, where 1 means equal.
Fraction of matching JSON leaf paths.
Sentence BLEU: n-grams 1 to 4 plus a brevity penalty.
ROUGE-L, the longest-common-subsequence F1.
Character n-gram F-score, tolerant of word order.
Token-overlap F1 between output and reference.
1 if the output parses as JSON.
1 if the output contains a given substring.
1 if the output matches a given regex.
Fraction of given items present in the output.
Built-in LLM judges
Ready-made judges you attach by name, no prompt to write. Each returns 0 to 1 where higher is better; for the safety judges, higher means safer. Grading uses claude-sonnet-5 by default, overridable per scorer. docs →
Answer versus reference: subset, superset, equal or disagree.
Is the answer correct given the reference? Graded 0 to 1.
Binary CORRECT or INCORRECT verdict; high precision on clear-cut errors.
Does the response cover every information need in the query? Catches partial answers.
Is the response well organized and easy to use, regardless of correctness?
Does the response address the request?
Is the response tight and free of filler?
Safety: 1 means free of toxic or harmful content.
Privacy: 1 means no PII disclosed.
Security: 1 means the run resisted prompt injection.
Pairwise: output versus reference in random A/B order to cancel positional bias. 1 win, 0.5 tie, 0 loss.
RAG & grounding judges
For retrieval-augmented answers: pass the retrieved passages as context and these grade grounding and retrieval quality. docs →
1 means every claim is grounded in the context. Holistic, one call.
Decomposes the answer into atomic claims and scores the fraction supported: your hallucination rate, with per-claim verdicts.
Is the retrieved context relevant to the question?
Does the context contain what the reference answer needs?
Is the answer responsive to the question?
Are the sources authoritative, relevant and current? Flags SEO spam and outdated citations.
Retrieval ranking & search metrics
Deterministic IR metrics over ranked retrieved ids, scored against binary relevance or graded qrels labels, plus model-free search checks. All accept an optional k cutoff. docs →
hit@k: 1 if any top-k id is relevant.
Reciprocal rank of the first relevant id.
Fraction of the top-k that are relevant.
Fraction of all relevant refs found in the top-k.
nDCG@k with graded gains when labels are set, else binary.
Per-query average precision, the term of MAP.
Fraction of the query's entities (proper nouns, quoted spans, numbers) that appear in the response.
Unique source domains behind the response, normalized 0 to 1 by a cap argument.
Embedding scorers
Semantic similarity via your project's embedding provider (OpenAI, Cohere or Gemini). Cosine similarity rescaled into 0 to 1; embedding tokens and cost recorded per score. docs →
Semantic closeness between output and expected, for when wording differs but meaning should match.
Query versus response similarity: catches answers that drift off the question.
Response versus sources similarity: a fast hallucination proxy.
Online eval kinds
Applied to live production traffic, scoped to all runs, one agent, or any metadata attribute, with a coverage slider to sample a percentage of matching runs. docs →
A model grades the run's output or its tool calls, with the judge model of your choice.
Model-free score of how well the agent used its tools, with optional recovery invalidation.
Property checks on JSON answers, plus shape-drift alerting via structured_output.changed.
Flags abnormal runtime, cost or span count per agent: the run baselines online eval.
Composite & human methods
For when one number is not enough: weighted multi-criterion scoring, dataset comparison, and human review that calibrates the judges. docs →
Multi-criterion scoring: each weighted dimension graded 1 to 5 in one call, with optional per-dimension minimums for pass or fail.
Compare outputs against expected values from a versioned dataset.
Human review queues write scores next to eval scores, and the judge-alignment table measures how often each judge agrees with your reviewers.
Or bring your own judge.
Custom judge prompts, weighted rubrics and your own goal signals all land as first-class scores next to these.