Code Evaluators

Code evaluators score your GenAI spans by running a short Python function. They are deterministic, repeatable, and free to run (no LLM call), which makes them a good fit for structural checks: JSON validity, required fields, regex and keyword matches, exact match, and length or format rules.

Each evaluation runs in an isolated microVM, so the result comes back in milliseconds and your code never touches other tenants' data.

Three ways to start

You want to	Use	Code to write
Run a standard check, such as keywords, PII or JSON validity	A built-in code check	None. Set its settings on the evaluator
Combine standard checks with your own rules	The oodle_eval library in your own template	A few lines
Score something that is special to your product	Your own evaluate function	All of it

All three run in the same sandbox and write the same kind of scores. You can also put code that many evaluators need into a shared library, give a template settings so that one template serves many evaluators, and read the scores of other evaluators on the same span. The cookbook has complete examples.

The evaluate Contract

Your source code defines one function, evaluate(ctx), which returns an EvaluationResult holding one or more Score objects. EvaluationResult, Score, and ctx are provided by the runtime, so there is nothing to import.

Classes of your own with those names also work, which is what an evaluator that must run outside Oodle too needs. The runtime reads the fields the table below lists, and takes the default for each one your class does not carry.

def evaluate(ctx):
    output = str(ctx.observation.output or "")

    return EvaluationResult(scores=[
        Score(
            name="has_output",
            value=len(output) > 0,
            data_type="BOOLEAN",
        ),
    ])

The ctx object

Field	Value
ctx.observation.input	The span's gen_ai.input.messages tag. Parsed JSON (a list of messages) when the tag holds JSON, otherwise the raw string
ctx.observation.output	The span's gen_ai.output.messages tag. Parsed JSON when the tag holds JSON, otherwise the raw string
ctx.observation.metadata	Every other span tag, as a dict of tag name to value. For example gen_ai.request.model or your own attributes. In an experiment it also carries dataset.expected_output, the item's expected output as text, when the item has one
ctx.trace.spans	The agent's steps, in order, for an experiment item. Each step is a dict with name, kind (llm, tool, agent or span), tool_name, model, input, output, duration_ms and error. An empty list for a live span
ctx.trace.tools_called()	The tool name of each tool step, in order
ctx.params	The evaluator's settings, as a dict. Every setting that the template declares is a key. An empty dict when the template declares none
ctx.scores	The scores that other evaluators wrote on the same span, as a dict of evaluator name to a list of scores. An empty dict when the code reads none
ctx.score(name)	The value of the first score that the evaluator called name wrote on this span, or None
ctx.experiment	The dataset item, in an experiment: item_id, expected_output, item_metadata (a dict), run_id and dataset_id. On a live span every field is None and item_metadata is {}
ctx.expected_output	Short for ctx.experiment.expected_output: the item's expected output, as the dataset stores it (a string or any JSON value). None on a live span and on an item without one

Both input and output are None when the span carries no value for that tag, so coerce before use (str(ctx.observation.output or "")). The text helpers read the reply, the user turns and the tool calls out of every common message shape for you.

Score

Argument	Required	Description
value	yes	The score. A float for NUMERIC, True / False for BOOLEAN, a string for CATEGORICAL
name	no	Score name shown on the trace. Defaults to the evaluator name. Use it when one evaluator returns several scores
data_type	no	NUMERIC (default), BOOLEAN, or CATEGORICAL
comment	no	Free-text explanation stored alongside the score
higher_is_better	no	Which way the score reads: True when a high value is good, False when a low value is good. It belongs on the score, not on the evaluator, because one evaluator can return both kinds. Leave it out on a categorical score, or to keep the evaluator's own setting

A NUMERIC score must hold a number. Text needs data_type="CATEGORICAL" and true/false needs "BOOLEAN": a numeric score holding text fails the evaluation with INVALID_RESULT and scores nothing.

What evaluate can return

• EvaluationResult(scores=[...]) with a list of Score objects.
• A dict with a non-empty scores list, for example {"scores": [{"name": "ok", "value": True, "data_type": "BOOLEAN"}]}.
• The list that a library check returns, or several of them added together: EvaluationResult(scores=metrics.refusal(ctx) + [mine]).

Return as many scores as you like from one evaluator. Give each score its own name:

def evaluate(ctx):
    output = str(ctx.observation.output or "")

    return EvaluationResult(scores=[
        Score(
            name="output_length",
            value=min(1.0, len(output) / 500),
            comment=f"Length: {len(output)} chars",
        ),
        Score(
            name="size_bucket",
            value=(
                "short" if len(output) < 100
                else "medium" if len(output) < 500
                else "long"
            ),
            data_type="CATEGORICAL",
        ),
    ])

Use the oodle_eval library

The oodle_eval library is in the sandbox. Import it like any module:

from oodle_eval.v1 import metrics, text, combine


def evaluate(ctx):
    # Two built-in checks. Each returns a list of scores.
    scores = metrics.conversation_degeneration(ctx)
    scores += metrics.pii_leak(ctx)

    # Your own score, from the reply text.
    reply = text.reply(ctx)
    mine = Score(
        name="mentions_refund",
        value="refund" in reply.lower(),
        data_type="BOOLEAN",
        higher_is_better=True,
    )

    return EvaluationResult(scores=scores + [mine])

Module	What it holds
metrics	The built-in checks as functions. Each takes ctx (or a plain string) and keyword settings, and returns a list of scores
text	Helpers that read the reply, the user turns, the tool calls, JSON and span attributes out of ctx
combine	Functions that make one score from several: weighted mean, all pass, threshold, and more
util	The helpers that the checks are built from, for checks of your own

The checks live in one file for each kind (metrics/format.py, quality, safety, conversation, similarity, context, agent and trace). Import them all from oodle_eval.v1.metrics.

The list that a check returns also finds a score by its name: scores.get("pii_free"), scores.value("pii_free") and scores.names().

Everything under oodle_eval.v1 keeps its behaviour. A later release changes it only to fix a bug, so your scores do not move when Oodle updates the library. The library reference lists every function, its arguments and the score names it returns. The code editor completes these names as you type.

Examples

Valid JSON output

import json


def evaluate(ctx):
    output = str(ctx.observation.output or "")
    try:
        json.loads(output)
        valid = True
    except ValueError:
        valid = False

    return EvaluationResult(scores=[
        Score(
            name="valid_json",
            value=valid,
            data_type="BOOLEAN",
        ),
    ])

Refusal detection

import re

REFUSAL = re.compile(
    r"\b(i can't|i cannot|i'm unable to)\b",
    re.IGNORECASE,
)


def evaluate(ctx):
    output = str(ctx.observation.output or "")
    match = REFUSAL.search(output)

    return EvaluationResult(scores=[
        Score(
            name="refused",
            value=bool(match),
            data_type="BOOLEAN",
            comment=match.group(0) if match else "",
        ),
    ])

Reading span metadata

def evaluate(ctx):
    meta = ctx.observation.metadata or {}
    model = str(meta.get("gen_ai.request.model", ""))

    return EvaluationResult(scores=[
        Score(
            name="model",
            value=model or "unknown",
            data_type="CATEGORICAL",
        ),
    ])

More examples are in the cookbook.

Creating a Code Evaluator

1. Go to Agent Observability → Evaluators (ap1, us1)
2. Click Create Evaluator → New Template
3. Select Code Evaluator
4. Name the evaluator. Use a folder/name prefix (for example safety/toxicity) to group related evaluators
5. Write your evaluate function in the editor, or pick a starter. A starter is a working example that you can change. Add settings if you want to set values per evaluator
6. Pick a span from Sample spans and click Run to execute the code against real data. The result panel shows the returned scores, or the error and any print() output if the run fails
7. Click Save

Then create an evaluation rule from the template to decide which spans it scores:

1. Evaluator Name is the score name that appears on traces
2. Settings are the values for the template's settings, if it has any
3. Filter rules restrict evaluation to spans matching span labels
4. Sampling rate is the percentage of matching spans to evaluate (1 to 100, default 100)
5. Max invocations/hour caps the run rate (set to 0 for no limit)
6. Dependencies run this rule only on spans that passed earlier rules. See Evaluator Dependencies

Because code evaluators are fast and cost nothing per run, they work well at 100% sampling, and as a level-0 filter that gates a more expensive LLM-as-Judge rule.

In experiments

In an experiment, ctx.expected_output holds the dataset item's expected output, and ctx.trace.spans holds the agent's steps. The checks that compare with a reference use the expected output when you do not set reference, so a template that runs metrics.exact_match(ctx) scores each item against its own expected output:

from oodle_eval.v1 import metrics


def evaluate(ctx):
    return EvaluationResult(scores=metrics.exact_match(ctx, normalize=True))

On a live span there is no expected output, so set the reference setting there.

Code evaluators also run in experiments. An experiment that includes a code evaluator or a built-in code check needs the same account access as creating one.

Execution Environment

Each span is evaluated in its own subprocess inside a Firecracker microVM, with a 5 second time limit and 128 MB of memory. The code has no network access. The runtime is pure Python with the standard library modules that are useful for scoring:

abc, array, base64, binascii, bisect, collections, contextlib, copy, dataclasses, datetime, decimal, difflib, enum, fractions, functools, hashlib, heapq, hmac, itertools, json, math, numbers, operator, pprint, random, re, statistics, string, struct, textwrap, typing, unicodedata, and for parsing output: csv, html, ipaddress, tomllib and urllib.parse (only parse: the other parts of urllib open network connections)

It also has oodle_eval (the library above) and shared (your team's shared libraries).

Use absolute imports, such as from oodle_eval.v1 import text. The sandbox checks each import by its top-level module name, so a relative import (from . import x) fails with BLOCKED_IMPORT.

Anything your code prints is captured and returned with the result, which makes print() the way to debug a test run.

Automate with the CLI and MCP

The oodle CLI and the Oodle MCP server do the same operations as the UI. For example, with the CLI:

# Starters and built-in checks, with their settings
oodle genai templates starters

# The oodle_eval library reference
oodle genai library

# Shared libraries
oodle genai code-libraries list
oodle genai code-libraries get <id>

When you create an evaluation rule with the CLI, the API or the MCP server, its filters use the same form as the UI:

[{"name": "span::gen_ai.operation.name", "type": "eq", "value": "chat"}]

• Name an attribute with its kind: span:: for a span attribute, resource:: for a resource attribute (for example resource::service.name). A dotted name without it is refused, because it would match no span.
• type is eq, neq, re, nre, oneof, not_oneof, gt, gte, lt or lte. oneof and not_oneof take a multi_value list instead of value. The API stores the type as the number the UI uses (eq as 0), so a rule reads back with the number.

An AI agent that uses the Oodle MCP server has two tools. genai_evaluator_workbench reads, tries and checks evaluators and changes nothing, so every role can use it. manage_genai_evaluators creates, changes and deletes them, and needs the Editor or Admin role. Each tool answers {"action": "help", "topic": "code"} with the guidance for a topic.

The agent can read the code that an evaluator can import, as you would read a codebase before you reuse it: the oodle_eval files and your team's shared libraries.

{"action": "grep_library", "pattern": "def .*refund", "ignore_case": true}
{"action": "library_files", "prefix": "oodle_eval/v1/metrics/"}
{"action": "read_library_file", "path": "shared/acme_text.py", "start_line": 1, "end_line": 80}
{"action": "code_library_reference", "function": "metrics.safety.pii_leak", "source": true}

grep_library searches with a regular expression and gives the file, the line number and the line. read_library_file gives up to 200 numbered lines, and reads a shared library at an earlier version too.

A typical run, in order:

1. Search for a check or a helper that already does the work (grep_library, code_library_reference).
2. Import it: a check from oodle_eval.v1.metrics, a helper from oodle_eval.v1.util.
3. Put code that several evaluators need in a shared library, and import it with from shared.<name> import ....
4. Try the draft, with any changed library, before you save it.
5. Pin library versions on the template where the scores must not move when a library changes.

{"action": "list_code_starters"}
{"action": "sample_spans", "filters": {"agent_name": "support"}, "start_time": "now-24h"}
{"action": "validate_code", "sourceCode": "...", "paramSpecs": [...]}
{"action": "try_code", "sourceCode": "from shared.acme_text import clean ...",
 "libraries": {"acme_text": "def clean(s): ..."}}

These go to genai_evaluator_workbench. Then, with manage_genai_evaluators, save the template and create the evaluator paused:

{"action": "create_template", "name": "Refund policy", "type": "code",
 "sourceCode": "...", "params": [{"name": "required", "type": "string_list"}]}
{"action": "create_evaluator", "name": "Refund policy", "templateId": "<template id>",
 "params": {"required": ["refund"]}, "enabled": false}

Try the saved evaluator on past spans, turn it on, and read its health after it has run for a while:

{"action": "try_code", "evaluatorId": "Refund policy", "start_time": "now-7d"}
{"action": "update_evaluator", "name": "Refund policy", "enabled": true}
{"action": "evaluator_health", "evaluator": "Refund policy", "start_time": "now-24h"}

try_code runs a draft, a starter, a template or an evaluator on real spans and saves nothing. It takes spans that you name (spanIds from sample_spans, or traceIds), a span that you paste (spanData, with its attributes in tags), or samples spans from several traces that match filters. Each row gives the scores, or the error and the printed output when the code fails. evaluator_health gives an evaluator's run count, pass rate, value histogram, errors by class and example spans.

Troubleshooting

Error	What it means
BLOCKED_IMPORT	The code imports a module outside the list above, uses a relative import, or calls one of the dynamic-execution builtins (eval, exec, open, getattr). Rewrite using the available modules
USER_CODE_ERROR	The code raised an exception. The message carries the Python traceback, and logs carries anything printed before the failure
TIMEOUT	The evaluation exceeded 5 seconds. Look for unbounded loops or backtracking regexes
MEMORY_EXCEEDED	The evaluation exceeded 128 MB. Avoid materialising large intermediate lists
INVALID_RESULT	evaluate() returned something other than an EvaluationResult (or a dict with a non-empty scores list), a score is missing value, or a NUMERIC score holds a value that is not a number. The message names the score
LIBRARY_NOT_FOUND	The code imports a shared library that does not exist, or the template pins a version that does not exist. See Shared libraries

Run against a sample span in the editor before saving: the test path and the production path share the same runtime, so a run that succeeds there succeeds on live spans.