Datasets & Experiments

Datasets let you organize test cases for systematic evaluation of your LLM application. Each dataset contains input/expected-output pairs that you can run through an LLM with attached evaluators to measure quality at scale.

Navigate to Agent Observability → Datasets in the sidebar (ap1, us1).

Creating a Dataset

1. Click + New Dataset
2. Enter a name (e.g. sms-dataset-v1)
3. Optionally upload a CSV file to import items
4. Click Create

note

Creating and managing datasets requires the Editor or Admin role.

CSV Import

You can bulk-import items from a CSV file (up to 50 MB). The CSV must include an input column (exact case). Optional columns:

Column	Description
input	The user query or prompt (required)
expected_output	The ideal response
metadata	Additional context as JSON

A preview of the first few rows is shown before import. Partial imports are supported: if some rows fail, the successful rows are still added and a summary toast shows the error count.

Adding Items

From a dataset's detail page, click + New Item to add entries manually or via CSV.

Manual Entry

Add one or more rows with Input (required), Expected Output, and Metadata fields. Click + Add another row to add multiple items at once.

From a Trace

On any trace detail page, click Add to Dataset. Oodle splits the conversation into input and expected output and adds it to the dataset you select. This is useful for capturing real production examples as test cases.

Running Experiments

Experiments run your dataset items through an LLM and optionally score the outputs with evaluators.

1. Open a dataset and click Run Experiment
2. Configure the experiment:

Field	Description
LLM Connection	Which provider credentials to use (configure here)
Model	The model to run items against
Evaluators	One or more active evaluator rules to score outputs
Output comparers	Evaluators that score the model's output against each item's expected output
Prompt	Optional. A prompt template (by label or version) whose {{variables}} each item's input fills

3. Click Run Experiment

Evaluators vs. output comparers

The two pickers hold different kinds of evaluator, and an evaluator appears in exactly one of them.

An ordinary evaluator scores a generation on its own: is it helpful, is it toxic, is it grounded in the context it was given. It needs nothing but the request and the response, so it also runs against live production traffic.

An output comparer scores the generation against the dataset item's expected output: does the model's answer mean the same thing as the answer you recorded. That makes it a regression test rather than a quality score, and it only works where ground truth exists. Live traffic has none, so a comparer runs in experiments only.

Two managed comparers ship with Oodle:

Comparer	What it scores
Output Match	Whether the actual output is equivalent in meaning to the expected output
Output Completeness	What fraction of the expected content the actual output covers

You can also write your own: pick Output Comparer when creating an evaluator template. An experiment fills exactly three variables for it:

Variable	Filled with
{{input}}	the dataset item's input
{{expected_output}}	the dataset item's expected output
{{output}}	what the model produced

Any other {{variable}} is left in the prompt as written, so the judge is handed the placeholder text rather than a value.

An output comparer has no filters, sampling rate, hourly cap or variable mapping, and no Active/Paused state. Those all describe running against live traffic, which a comparer never does: it runs when an experiment names it, and Oodle supplies the three variables above from the dataset item.

note

An item with no expected output is skipped by every output comparer, not scored zero. The run reports how many comparisons it skipped, so a dataset that was never given ground truth does not read as a failing model.

How the item input reaches the model

The Prompt field decides how each item's input becomes the request.

With a prompt

The item's input supplies the template's {{variables}}:

Item input	What it fills
An object, {"question": "Where is my order?", "tone": "brief"}	Each key fills the variable of the same name: {{question}}, {{tone}}
A plain string, "Where is my order?"	Fills {{input}}
A message array, [{"role": "user", ...}]	Supplies no variables. Run conversation datasets without a prompt

So the template Answer in {{tone}}: {{question}} with the object input above sends Answer in brief: Where is my order?.

A dataset of plain strings works the same way through the one {{input}} variable: the template Answer briefly: {{input}} sends Answer briefly: Where is my order?. Use an object input when a template needs more than one value.

Matching is on the exact key, and a variable name is word characters only: {{question}} is a variable, {{user.name}} is literal text. A variable with no matching value stays in the prompt exactly as written, so {{user_name}} reaches the model as those characters and the reply usually talks about the placeholder rather than the item.

The Run Experiment form lists the selected prompt version's variables and marks the ones no item in the dataset supplies, so a mismatch is visible before you spend a run. Renaming a dataset key or a template variable so the two agree is what fixes it.

With no prompt

The item's input is the request: a message array is sent as chat messages, any other value as its JSON text. That is the shape for a dataset that already holds full prompts or captured conversations.

expected_output is never sent to the model either way. It reaches evaluators only, as the ground truth they compare the output against.

The experiment runs asynchronously. Results appear in the Runs tab.

tip

If you don't have an LLM connection configured yet, the form links to Settings → LLM Connections where you can add one.

Running on a Schedule

A dataset can run its experiment on its own, so a prompt or model change is caught by the next scheduled run rather than the next time someone remembers to press the button.

1. Open a dataset and go to the Schedule tab
2. Under When it runs, pick one of the two shapes:

At set times, for a run that has to land at a time of day you care about (before standup, after a nightly data load):

Field	Description
Days of week	Which days fire. Leave all unselected to run every day
Start times	One or more HH:MM times. Each one is a separate run over the whole dataset
Days of month	Optional. With days of week also set, a day has to match both
Timezone	The zone the start times are read in

On a period, for a run whose cadence matters but whose wall-clock time does not: every N minutes, hours or days. The shortest period is 5 minutes, and the first run happens one period after you save. No timezone applies, because a duration is the same length everywhere.

3. Under What each run does, configure the same connection, model, evaluators, output comparers and prompt the Run Experiment form takes
4. Click Create schedule

Each firing creates its own run under the Runs tab, numbered like a manual run. The tab shows the next and last run times, and if a scheduled run could not start (a deleted LLM connection, say) it shows why.

A run starts shortly after its scheduled time rather than exactly on the minute.

note

The next run time is set when a run is queued, not when it finishes, so a long or failed run never delays or stops the schedule.

If a schedule misses several runs (an outage, or the dataset's workers were down), it does not replay them. One catch-up run happens and the schedule then resumes its normal cadence.

note

Times are read in the schedule's timezone, so a 09:00 schedule stays at nine o'clock local across daylight saving changes rather than drifting by an hour.

Turn the Enabled switch off to stop a schedule firing while keeping its definition.

Viewing Results

The Runs tab shows all experiment runs with their status, LLM connection, evaluator, prompt reference, and creation time.

Click any run to open a detail drawer showing:

• Run metadata: status, connection, model, evaluators, prompt, and any errors
• Per-item results: a table with Input, Metadata, Expected Output, Actual Output, and score badges from each evaluator

While an experiment is running, the drawer polls for new results every few seconds and shows a progress indicator.

Run Statuses

Status	Meaning
Pending	Queued, not yet started
Running	Processing dataset items
Completed	All items processed
Failed	An error stopped the run