Code Evaluators

Code evaluators score your GenAI spans by running a short Python function. They are deterministic, repeatable, and free to run (no LLM call), which makes them a good fit for structural checks: JSON validity, required fields, regex and keyword matches, exact match, and length or format rules.

Each evaluation runs in an isolated microVM, so the result comes back in milliseconds and your code never touches other tenants' data.

The evaluate Contract

Your source code defines one function, evaluate(ctx), which returns an EvaluationResult holding one or more Score objects. EvaluationResult, Score, and ctx are provided by the runtime, so there is nothing to import.

def evaluate(ctx):
    output = str(ctx.observation.output or "")

    return EvaluationResult(scores=[
        Score(
            name="has_output",
            value=len(output) > 0,
            data_type="BOOLEAN",
        ),
    ])

ctx.observation

Field	Source span tag	Value
ctx.observation.input	gen_ai.input.messages	Parsed JSON (a list of messages) when the tag holds JSON, otherwise the raw string
ctx.observation.output	gen_ai.output.messages	Parsed JSON when the tag holds JSON, otherwise the raw string
ctx.observation.metadata	every other span tag	Dict of tag name to value, for example gen_ai.request.model or your own attributes

Both input and output are None when the span carries no value for that tag, so coerce before use (str(ctx.observation.output or "")).

Score

Argument	Required	Description
value	yes	The score. A float for NUMERIC, True / False for BOOLEAN, a string for CATEGORICAL
name	no	Score name shown on the trace. Defaults to the evaluator name. Use it when one evaluator returns several scores
data_type	no	NUMERIC (default), BOOLEAN, or CATEGORICAL
comment	no	Free-text explanation stored alongside the score

Return as many scores as you like from one evaluator:

def evaluate(ctx):
    output = str(ctx.observation.output or "")

    return EvaluationResult(scores=[
        Score(
            name="output_length",
            value=min(1.0, len(output) / 500),
            comment=f"Length: {len(output)} chars",
        ),
        Score(
            name="size_bucket",
            value=(
                "short" if len(output) < 100
                else "medium" if len(output) < 500
                else "long"
            ),
            data_type="CATEGORICAL",
        ),
    ])

Examples

Valid JSON output

import json


def evaluate(ctx):
    output = str(ctx.observation.output or "")
    try:
        json.loads(output)
        valid = True
    except ValueError:
        valid = False

    return EvaluationResult(scores=[
        Score(
            name="valid_json",
            value=valid,
            data_type="BOOLEAN",
        ),
    ])

Refusal detection

import re

REFUSAL = re.compile(
    r"\b(i can't|i cannot|i'm unable to)\b",
    re.IGNORECASE,
)


def evaluate(ctx):
    output = str(ctx.observation.output or "")
    match = REFUSAL.search(output)

    return EvaluationResult(scores=[
        Score(
            name="refused",
            value=bool(match),
            data_type="BOOLEAN",
            comment=match.group(0) if match else "",
        ),
    ])

Reading span metadata

def evaluate(ctx):
    meta = ctx.observation.metadata or {}
    model = str(meta.get("gen_ai.request.model", ""))

    return EvaluationResult(scores=[
        Score(
            name="model",
            value=model or "unknown",
            data_type="CATEGORICAL",
        ),
    ])

Creating a Code Evaluator

1. Go to Agent Observability → Evaluators (ap1, us1)
2. Click Create Evaluator → New Template
3. Select Code Evaluator
4. Name the evaluator. Use a folder/name prefix (for example safety/toxicity) to group related evaluators
5. Write your evaluate function in the editor. It is pre-filled with a working example
6. Pick a span from Sample spans and click Run to execute the code against real data. The result panel shows the returned scores, or the error and any print() output if the run fails
7. Click Save

Then create an evaluation rule from the template to decide which spans it scores:

1. Evaluator Name is the score name that appears on traces
2. Filter rules restrict evaluation to spans matching span labels
3. Sampling rate is the percentage of matching spans to evaluate (1 to 100, default 100)
4. Max invocations/hour caps the run rate (set to 0 for no limit)
5. Dependencies run this rule only on spans that passed earlier rules. See Evaluator Dependencies

Because code evaluators are fast and cost nothing per run, they work well at 100% sampling, and as a level-0 filter that gates a more expensive LLM-as-Judge rule.

Execution Environment

Each span is evaluated in its own subprocess inside a Firecracker microVM, with a 5 second time limit and 128 MB of memory. The runtime is pure Python with the standard library modules that are useful for scoring:

abc, array, base64, binascii, bisect, collections, contextlib, copy, dataclasses, datetime, decimal, difflib, enum, fractions, functools, hashlib, heapq, hmac, itertools, json, math, numbers, operator, pprint, random, re, statistics, string, struct, textwrap, typing, unicodedata

Anything your code prints is captured and returned with the result, which makes print() the way to debug a test run.

Troubleshooting

Error	What it means
BLOCKED_IMPORT	The code imports a module outside the list above, or calls one of the dynamic-execution builtins (eval, exec, open, getattr). Rewrite using the available modules
USER_CODE_ERROR	The code raised an exception. The message carries the Python traceback, and logs carries anything printed before the failure
TIMEOUT	The evaluation exceeded 5 seconds. Look for unbounded loops or backtracking regexes
MEMORY_EXCEEDED	The evaluation exceeded 128 MB. Avoid materialising large intermediate lists
INVALID_RESULT	evaluate() returned something other than an EvaluationResult (or a dict with a non-empty scores list), or a score is missing value

Run against a sample span in the editor before saving: the test path and the production path share the same runtime, so a run that succeeds there succeeds on live spans.