Code Evaluator Settings

A code template can declare settings: named values that its code reads from ctx.params. Each evaluator that runs the template sets its own values. One template then serves many evaluators, and you change a keyword list or a threshold without a copy of the code.

For example, one "Keyword check" template can run as two evaluators: one that requires refund on support replies, and one that forbids guarantee on sales replies.

Declare settings on a template

In the code template form, add a row for each setting to the Settings table below the code. Through the API, the CLI or MCP, send them as the template's params list:

{
  "name": "Keyword check",
  "type": "code",
  "sourceCode": "...",
  "params": [
    {
      "name": "required",
      "type": "string_list",
      "label": "Required keywords",
      "description": "The reply must mention each of these.",
      "default": []
    },
    {
      "name": "mode",
      "type": "enum",
      "options": ["all", "any"],
      "default": "all"
    },
    {
      "name": "case_sensitive",
      "type": "boolean",
      "default": false
    }
  ]
}

Field	Required	Description
name	yes	The key in ctx.params. A lower-case Python name: letters, digits and _, up to 64 characters, not starting with a digit
type	yes	One of the types below
label	no	The label on the evaluator form
description	no	Help text on the evaluator form
default	no	The value when an evaluator does not set one. It must have the setting's type
required	no	true makes each evaluator set a value, unless there is a default
options	for enum	The values an enum setting can have

A template can declare up to 32 settings, and each name only once.

Types

Type	Value in ctx.params	Form input
string	str	One line of text
text	str	Several lines of text
number	int or float	Number
integer	int (a whole number)	Number
boolean	True or False	Switch
string_list	list of str	One input for each value. A value can contain commas
enum	str, one of options	Drop-down
json	Any JSON value (dict, list, number, string)	JSON editor

Set values on an evaluator

When you create or edit an evaluator on a template with settings, the form shows one input for each setting. Through the API, the CLI or MCP, send the values as the evaluator's params object:

{
  "name": "Support replies mention refunds",
  "evaluatorId": "<template id>",
  "params": {"required": ["refund"], "mode": "any"}
}

Oodle checks the values against the template's settings when you save the evaluator. It refuses a value of the wrong type, a name that the template does not declare, and a missing required value.

Change the settings of a template

You can add, change and remove settings on a template that evaluators already use. Oodle refuses the change (HTTP 409) when an existing evaluator holds a value that the new settings would not accept: for example, a value for a setting that you remove, a value of a type that you change, or no value for a setting that you make required with no default. The error names the evaluators. Change their values first, then change the template.

What the code gets

ctx.params holds every setting that the template declares:

1. The evaluator's value, if it sets one.
2. Else the setting's default.
3. Else None.

So ctx.params["name"] never raises KeyError for a declared setting. Code that allows None can use ctx.params.get("name") or <fallback>.

from oodle_eval.v1 import metrics


def evaluate(ctx):
    return EvaluationResult(scores=metrics.keyword_check(ctx, **ctx.params))

When the setting names are the same as the keyword arguments of a library check, **ctx.params passes them all. The built-in code checks work this way.

Settings in tests and experiments

• A test run of a draft template uses the values that you give for the run, else the defaults.
• An experiment that runs an evaluator uses that evaluator's values.
• A test run of a draft checks the values against the settings that you send with it (paramSpecs). With MCP, give starterId or templateId to use the settings of a starter or a saved template. An experiment that runs a template with no evaluator uses the defaults.
• A change to a setting's default changes the value for every evaluator that does not set its own.

With the CLI and MCP

oodle genai templates starters                  # lists the setting names
oodle genai templates starters keyword-check    # a template file with each setting's type and default
oodle genai evaluators get "Refund mentioned"   # the values one evaluator sets

With MCP, manage_genai_evaluators sets values, and genai_evaluator_workbench tries them:

{"action": "update_evaluator", "name": "Refund mentioned",
 "params": {"required": ["refund", "return"]}}
{"action": "try_code", "sourceCode": "...",
 "paramSpecs": [{"name": "at", "type": "number", "default": 0.5}],
 "params": {"at": 0.7}}
{"action": "try_code", "starterId": "keyword-check",
 "params": {"required": ["refund"]}}