LLM-as-Judge Evaluators

LLM-as-Judge evaluators use a language model to score your GenAI spans. They are ideal for subjective assessments like relevance, helpfulness, and tone that are hard to express as deterministic code.

Built-in Templates

Oodle includes eight managed evaluator templates ready to use out of the box. Each returns a score from 0 to 1 with reasoning.

Template	Variables	What it measures
Hallucination	query, generation, context	Whether the output is grounded in the query and context
Helpfulness	query, generation	How helpful and complete the answer is
Relevance	query, generation	How relevant the output is to the query
Toxicity	generation	Presence of toxic content, hate speech, or profanity
Correctness	query, generation, ground_truth	Factual correctness against a known answer
Conciseness	query, generation	Brevity without losing completeness
Context Relevance	query, context	Whether retrieved context is relevant to the query (RAG)
Faithfulness	generation, context	Whether the output stays faithful to the context (RAG)

Managed templates cannot be edited or deleted.

Creating a Custom Template

1. Go to GenAI → Evaluators (ap1, us1)
2. Click Create Evaluator → New Template
3. Select LLM as a Judge
4. Enter a name and write your evaluation prompt using {{variable}} placeholders
5. Click Save

The template automatically appends return instructions that ask the model to respond with a JSON score (0–1) and reasoning.

Writing Effective Prompts

Use {{variable}} syntax to reference span data. Common variables:

• {{query}} — the user's input
• {{generation}} — the model's output
• {{context}} — system instructions or retrieved context

Example:

You are an expert evaluator.

Score the following output on a scale of 0 to 1
based on whether it directly answers the question.

User query: {{query}}
LLM output: {{generation}}

Judging with Jev

A rule whose connection is TypeSafe (Jev) runs the same template through Jev instead of a chat model. Jev takes the evidence and the question separately, so write the prompt as one yes/no question and refer to the variables by name:

Does {{generation}} answer {{query}} without
inventing facts?

Each {{variable}} is sent to Jev as a field, and the question reads Does generation answer query without inventing facts?.

The rule form shows the same Decision settings an experiment on Jev has, because the two are set up alike:

Setting	What it does	Score written
Decision: Yes / no	the question is answered with P(yes)	numeric, P(yes); keep Higher is better on when yes is the good answer and off when it is not (Is {{generation}} toxic?)
Decision: One of a set of options	Jev picks one of the options you list	categorical, the option
Decision: A level on an ordered scale	Jev places the answer on your levels	numeric, where on the scale it landed
Threshold (yes/no only)	P(yes) at or above which the answer reads "yes"; 0.5 unless set
Confidence	how sure Jev has to be for its answer to be final; 0.7 unless set

Jev also reports how sure it is. The score comment says the answer, or unsure. For dependent rules an unsure verdict is the finding and a confident one, either way, is final: a more expensive judge chained behind Jev sees only what Jev could not decide. Confidence is that line, on a 0–1 scale. Through the API the settings are modelParams.decision on the rule: {"type": "noul", "threshold": 0.15, "confidence": 0.8}.

Creating an Evaluation Rule

After selecting a template (built-in or custom), configure how it runs:

1. Name — the score name that appears on traces
2. LLM Connection — which provider to use for evaluation (configure here)
3. Model — which model runs the evaluation
4. Variable mapping — map template variables to span fields
5. Sampling rate — percentage of matching spans to evaluate (1–100%, default 100%)
6. Max invocations/hour — rate cap to control cost (default 100, set to 0 for no limit)
7. Filter rules — optional span label filters to restrict which spans are evaluated

Click Create to activate the rule.

Variable Mapping

Each {{variable}} in the template prompt maps to a span field. Oodle auto-assigns sensible defaults:

Variable	Default field
query, input	gen_ai.input.messages
generation, output	gen_ai.output.messages
context, system	gen_ai.system_instructions
ground_truth	expected_output

You can override any mapping by selecting a different span tag from the dropdown. Add an optional JsonPath expression (e.g. $[0].content) to extract a nested value from JSON fields.

Use the Preview toggle with a sample span to verify that variables resolve correctly before creating the rule.

Managing Evaluators

Evaluators Tab

The main tab shows all active evaluation rules. Each row displays the rule name, status (Active / Paused), 24-hour cost, and timestamps.

Click a rule to open a detail drawer with:

• Score over time chart (1h / 6h / 24h / 7d / 30d)
• Execution logs with per-span score, reasoning, and trace links
• Configuration summary (connection, model, template, mappings)

Library Tab

Browse all templates — managed and custom. Click any template to see its full prompt text, variables, and the list of rules using it. Use Use this evaluator to create a new rule from the template.

Scores Tab

View all scores produced by evaluators across a configurable time range. Filter by score name, value range, and span labels. Click any row to open the trace detail.