LLM-as-Judge Evaluators

LLM-as-Judge evaluators use a language model to score your GenAI spans. They are ideal for subjective assessments like relevance, helpfulness, and tone that are hard to express as deterministic code.

Built-in Templates

Oodle includes eight managed evaluator templates ready to use out of the box. Each returns a score from 0 to 1 with reasoning.

Template	Variables	What it measures
Hallucination	query, generation, context	Whether the output is grounded in the query and context
Helpfulness	query, generation	How helpful and complete the answer is
Relevance	query, generation	How relevant the output is to the query
Toxicity	generation	Presence of toxic content, hate speech, or profanity
Correctness	query, generation, ground_truth	Factual correctness against a known answer
Conciseness	query, generation	Brevity without losing completeness
Context Relevance	query, context	Whether retrieved context is relevant to the query (RAG)
Faithfulness	generation, context	Whether the output stays faithful to the context (RAG)

Managed templates cannot be edited or deleted.

Creating a Custom Template

1. Go to GenAI → Evaluators (ap1, us1)
2. Click Create Evaluator → New Template
3. Select LLM as a Judge
4. Enter a name and write your evaluation prompt using {{variable}} placeholders
5. Click Save

The template automatically appends return instructions that ask the model to respond with a JSON score (0–1) and reasoning.

Writing Effective Prompts

Use {{variable}} syntax to reference span data. Common variables:

• {{query}} — the user's input
• {{generation}} — the model's output
• {{context}} — system instructions or retrieved context

Example:

You are an expert evaluator.

Score the following output on a scale of 0 to 1
based on whether it directly answers the question.

User query: {{query}}
LLM output: {{generation}}

Creating an Evaluation Rule

After selecting a template (built-in or custom), configure how it runs:

1. Name — the score name that appears on traces
2. LLM Connection — which provider to use for evaluation (configure here)
3. Model — which model runs the evaluation
4. Variable mapping — map template variables to span fields
5. Sampling rate — percentage of matching spans to evaluate (1–100%, default 100%)
6. Max invocations/hour — rate cap to control cost (default 100, set to 0 for no limit)
7. Filter rules — optional span label filters to restrict which spans are evaluated

Click Create to activate the rule.

Variable Mapping

Each {{variable}} in the template prompt maps to a span field. Oodle auto-assigns sensible defaults:

Variable	Default field
query, input	gen_ai.input.messages
generation, output	gen_ai.output.messages
context, system	gen_ai.system_instructions
ground_truth	expected_output

You can override any mapping by selecting a different span tag from the dropdown. Add an optional JsonPath expression (e.g. $[0].content) to extract a nested value from JSON fields.

Use the Preview toggle with a sample span to verify that variables resolve correctly before creating the rule.

Managing Evaluators

Evaluators Tab

The main tab shows all active evaluation rules. Each row displays the rule name, status (Active / Paused), 24-hour cost, and timestamps.

Click a rule to open a detail drawer with:

• Score over time chart (1h / 6h / 24h / 7d / 30d)
• Execution logs with per-span score, reasoning, and trace links
• Configuration summary (connection, model, template, mappings)

Library Tab

Browse all templates — managed and custom. Click any template to see its full prompt text, variables, and the list of rules using it. Use Use this evaluator to create a new rule from the template.

Scores Tab

View all scores produced by evaluators across a configurable time range. Filter by score name, value range, and span labels. Click any row to open the trace detail.