Datasets & Experiments

Datasets let you organize test cases for systematic evaluation of your LLM application. Each dataset contains input/expected-output pairs that you can run through an LLM with attached evaluators to measure quality at scale.

Navigate to Agent Observability → Datasets in the sidebar (ap1, us1).

Creating a Dataset

1. Click + New Dataset
2. Enter a name (e.g. sms-dataset-v1)
3. Optionally upload a CSV file to import items
4. Click Create

note

Creating and managing datasets requires the Editor or Admin role.

CSV Import

You can bulk-import items from a CSV file (up to 50 MB). The CSV must include an input column (exact case). Optional columns:

Column	Description
input	The user query or prompt (required)
expected_output	The ideal response
metadata	Additional context as JSON

A preview of the first few rows is shown before import. Partial imports are supported: if some rows fail, the successful rows are still added and a summary toast shows the error count.

Adding Items

From a dataset's detail page, click + New Item to add entries manually or via CSV.

Manual Entry

Add one or more rows with Input (required), Expected Output, and Metadata fields. Click + Add another row to add multiple items at once.

From a Trace

On any trace detail page, click Add to Dataset. Oodle splits the conversation into input and expected output and adds it to the dataset you select. This is useful for capturing real production examples as test cases.

Running Experiments

Experiments run your dataset items through a target and optionally score the outputs with evaluators. The target is either an LLM connection (a model, with an optional prompt) or a webhook: an endpoint you host that runs your own agent or workflow. See Testing your own agent through a webhook.

1. Open a dataset and click Run Experiment
2. Under What to run, pick A model with a prompt or Your own agent
3. Configure the experiment:

Field	Description
LLM Connection	Which provider credentials to use (configure here)
Webhook	On the webhook tab: the endpoint each item is posted to. Create or test one inline, or manage them under Agent Observability → Settings → Experiment Webhooks
Evaluator LLM connection	On the webhook tab: the connection LLM judges score with when no selected evaluator names its own
Model	The model to run items against
Evaluators	One or more active evaluator rules to score outputs
Output comparers	Evaluators that score the model's output against each item's expected output
Prompt	Optional. A prompt template (by label or version) whose {{variables}} each item's input fills

3. Click Run Experiment

Evaluators vs. output comparers

The two pickers hold different kinds of evaluator, and an evaluator appears in exactly one of them.

An ordinary evaluator scores a generation on its own: is it helpful, is it toxic, is it grounded in the context it was given. It needs nothing but the request and the response, so it also runs against live production traffic.

An output comparer scores the generation against the dataset item's expected output: does the model's answer mean the same thing as the answer you recorded. That makes it a regression test rather than a quality score, and it only works where ground truth exists. Live traffic has none, so a comparer runs in experiments only.

Two managed comparers ship with Oodle:

Comparer	What it scores
Output Match	Whether the actual output is equivalent in meaning to the expected output
Output Completeness	What fraction of the expected content the actual output covers

You can also write your own: pick Output Comparer when creating an evaluator template. An experiment fills exactly three variables for it:

Variable	Filled with
{{input}}	the dataset item's input
{{expected_output}}	the dataset item's expected output
{{output}}	what the model produced

Any other {{variable}} is left in the prompt as written, so the judge is handed the placeholder text rather than a value.

An output comparer has no filters, sampling rate, hourly cap or variable mapping, and no Active/Paused state. Those all describe running against live traffic, which a comparer never does: it runs when an experiment names it, and Oodle supplies the three variables above from the dataset item.

note

An item with no expected output is skipped by every output comparer, not scored zero. The run reports how many comparisons it skipped, so a dataset that was never given ground truth does not read as a failing model.

How the item input reaches the model

The Prompt field decides how each item's input becomes the request.

With a prompt

The item's input supplies the template's {{variables}}:

Item input	What it fills
An object, {"question": "Where is my order?", "tone": "brief"}	Each key fills the variable of the same name: {{question}}, {{tone}}
A plain string, "Where is my order?"	Fills {{input}}
A message array, [{"role": "user", ...}]	Supplies no variables. Run conversation datasets without a prompt

So the template Answer in {{tone}}: {{question}} with the object input above sends Answer in brief: Where is my order?.

A dataset of plain strings works the same way through the one {{input}} variable: the template Answer briefly: {{input}} sends Answer briefly: Where is my order?. Use an object input when a template needs more than one value.

Matching is on the exact key, and a variable name is word characters only: {{question}} is a variable, {{user.name}} is literal text. A variable with no matching value stays in the prompt exactly as written, so {{user_name}} reaches the model as those characters and the reply usually talks about the placeholder rather than the item.

The Run Experiment form lists the selected prompt version's variables and marks the ones no item in the dataset supplies, so a mismatch is visible before you spend a run. Renaming a dataset key or a template variable so the two agree is what fixes it.

With no prompt

The item's input is the request: a message array is sent as chat messages, any other value as its JSON text. That is the shape for a dataset that already holds full prompts or captured conversations.

expected_output is never sent to the model either way. It reaches evaluators only, as the ground truth they compare the output against.

The experiment runs asynchronously. Results appear in the Runs tab.

tip

If you don't have an LLM connection configured yet, the form links to Agent Observability → Settings → LLM Connections where you can add one.

Testing your own agent through a webhook

An LLM connection tests one model call. A webhook tests the whole workflow: Oodle posts each dataset item to an endpoint you host, your agent does whatever it does with it, and the reply is scored the same way a model's would be. Every request carries a W3C traceparent header, so each result row opens the trace your agent emitted for that item.

Pick Your own agent on the Run Experiment form and choose a webhook, or create one inline. A run against a webhook has no generation connection, so the LLM judges and output comparers you pick need one of their own: an evaluator that names its own connection uses it, and the Evaluator LLM connection field covers any that do not.

The evaluators can read the trace as well as the answer: a judge prompt takes {{trajectory}}, and a code evaluator gets ctx.trace.spans, so a rule can assert on the tools the agent called and not only on what it said.

See Experiment Webhooks for every parameter, the request template placeholders, the trace check, trajectory evaluation, and the CLI, Terraform and MCP equivalents.

Alerting on experiment scores

Every run pushes what its scorers said to your instance as metrics, tagged with the dataset, so a monitor can watch an output comparer or an evaluator without anyone opening the Runs tab. They are delta counters: each run adds its own sum and count per score.

Metric	Labels	Meaning
oodle_genai_experiment_score_sum	dataset_id, dataset_name, score_name, scorer, target, data_type	Sum of the score values a run wrote. A boolean counts as 0 or 1
oodle_genai_experiment_score_count	same, plus category for a categorical score	How many scores were written
oodle_genai_experiment_score_min, _max	same as the sum	Gauges: the lowest and highest value one item scored in the run
oodle_genai_experiment_score_failures_total	dataset_id, dataset_name, score_name, scorer, target	Scorers that raised and wrote no score
oodle_genai_experiment_items_total	dataset_id, dataset_name, target, status	Items that completed or failed

scorer is output_comparer or evaluator; target is llm or webhook; score_name is the evaluator's name.

The mean score over the runs of the last day, per dataset and scorer:

sum by (dataset_name, score_name) (
  increase(oodle_genai_experiment_score_sum[1d])
)
/
sum by (dataset_name, score_name) (
  increase(oodle_genai_experiment_score_count[1d])
)

A monitor on that expression with a threshold such as < 0.9 for scorer="output_comparer" fires when the comparer's pass rate on a dataset drops. Set the window to cover at least one scheduled run, or the expression has nothing to read between runs. The share of a categorical verdict is the same ratio with category="<verdict>" on the numerator.

A mean hides the one item that scored zero. To fire on any single item, read the run's low instead:

min_over_time(
  oodle_genai_experiment_score_min{dataset_name="support-faq"}[1d]
) < 0.5

Running on a Schedule

A dataset can run its experiment on its own, so a prompt or model change is caught by the next scheduled run rather than the next time someone remembers to press the button.

1. Open a dataset and go to the Schedule tab
2. Under When it runs, pick one of the two shapes:

At set times, for a run that has to land at a time of day you care about (before standup, after a nightly data load):

Field	Description
Days of week	Which days fire. Leave all unselected to run every day
Start times	One or more HH:MM times. Each one is a separate run over the whole dataset
Days of month	Optional. With days of week also set, a day has to match both
Timezone	The zone the start times are read in

On a period, for a run whose cadence matters but whose wall-clock time does not: every N minutes, hours or days. The shortest period is 5 minutes, and the first run happens one period after you save. No timezone applies, because a duration is the same length everywhere.

3. Under What each run does, configure the same connection, model, evaluators, output comparers and prompt the Run Experiment form takes
4. Click Create schedule

Each firing creates its own run under the Runs tab, numbered like a manual run. The tab shows the next and last run times, and if a scheduled run could not start (a deleted LLM connection, say) it shows why.

A run starts shortly after its scheduled time rather than exactly on the minute.

note

The next run time is set when a run is queued, not when it finishes, so a long or failed run never delays or stops the schedule.

If a schedule misses several runs (an outage, or the dataset's workers were down), it does not replay them. One catch-up run happens and the schedule then resumes its normal cadence.

note

Times are read in the schedule's timezone, so a 09:00 schedule stays at nine o'clock local across daylight saving changes rather than drifting by an hour.

Turn the Enabled switch off to stop a schedule firing while keeping its definition.

Viewing Results

The Runs tab shows all experiment runs with their status, LLM connection, evaluator, prompt reference, and creation time.

Click any run to open a detail drawer showing:

• Run metadata: status, connection, model, evaluators, prompt, and any errors
• Per-item results: a table with Input, Metadata, Expected Output, Actual Output, and score badges from each evaluator

While an experiment is running, the drawer polls for new results every few seconds and shows a progress indicator.

Run Statuses

Status	Meaning
Pending	Queued, not yet started
Running	Processing dataset items
Completed	All items processed
Failed	An error stopped the run