Experiments API

An experiment is a set of rows. Each row is one input that your code ran, the output it gave, and the scores it got. Oodle keeps the experiment, links each row to its GenAI trace, and compares the experiment with earlier ones.

Use this API when you run evals in your own code: on a laptop, on an EC2 host, or in a GitHub Actions job. Oodle does not need to reach your code. Your code calls Oodle.

To run a dataset through a model or your agent from Oodle, see Datasets & Experiments. If your code uses the Braintrust SDK, see Migrate from Braintrust.

Base URL and authentication

All paths on this page are under:

https://<API_DOMAIN>/v1/api/instance/<OODLE_INSTANCE>/langfuse/api/public/

Send your Oodle API key in the X-API-KEY header. The examples use these variables:

export OODLE_API="https://<API_DOMAIN>/v1/api/instance/<OODLE_INSTANCE>/langfuse/api/public"
export OODLE_API_KEY="<OODLE_API_KEY>"

Endpoints

Method and path	Purpose
POST experiments	Create an experiment, or get the one with the same project and name
GET experiments	List experiments. Query: projectId (* for all projects), datasetId, name, source, page, limit
GET experiments/{id}	Get one experiment
PATCH experiments/{id}	Change name, description, metadata, tags or baseExperimentId
DELETE experiments/{id}	Delete the experiment, its rows and their scores
POST experiments/{id}/rows	Insert or update rows
GET experiments/{id}/rows	Read rows with their scores. Query: limit, cursor
POST experiments/{id}/feedback	Add scores, expected values, tags, metadata or comments to rows
GET experiments/{id}/summary	Score and metric averages, compared with a base experiment
GET experiment-comparisons	Rows of two to five experiments, matched by input

Oodle keeps an experiment, with its rows and their scores, for 60 days after it was created. Its traces follow the trace retention of your instance.

Create an experiment

curl -X POST "$OODLE_API/experiments" \
  -H "X-API-KEY: $OODLE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "<PROJECT_ID>",
    "name": "prompt-v2",
    "datasetId": "<DATASET_ID>",
    "parameters": {"model": "gpt-4.1", "prompt_version": 7},
    "tags": ["ci"],
    "repoInfo": {"commit": "3f2c1ab", "branch": "feature/new-prompt"}
  }'

Field	Description
projectId	The project. Omit it for the default project
name	The experiment name. Create is idempotent by project and name: a second create with the same name returns the first experiment (status 200). A new experiment returns 201
ensureNew	true makes a new experiment each time. When the name is in use, Oodle adds a number to it
datasetId	Optional. The dataset that the rows came from
datasetVersion	Optional. The RFC 3339 time of the dataset version that you read
baseExperimentId	Optional. The experiment that the summary compares with
parameters	The settings of the run, for example the model and the prompt version
metadata, tags, description	Your own data
repoInfo	The git state of the run: commit, branch, tag, dirty, author_name, commit_message, commit_time

The reply holds the experiment id.

Insert rows

Send rows in batches. Each row gives the input, the output and the scores of one case.

curl -X POST "$OODLE_API/experiments/<EXPERIMENT_ID>/rows" \
  -H "X-API-KEY: $OODLE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rows": [
      {
        "id": "case-0001",
        "input": {"question": "How do I reset my password?"},
        "output": "Open Settings, then Security, then Reset password.",
        "expected": "Settings > Security > Reset password",
        "scores": {"correctness": 1, "concise": 0.8},
        "metrics": {"prompt_tokens": 412, "completion_tokens": 37, "estimated_cost": 0.0021},
        "metadata": {"category": "account"}
      }
    ]
  }'

The reply gives the stored row ids in the order of the request: {"rowIds": ["case-0001"]}.

Field	Description
id	Your id for the row. A write with the same id updates the row, so you can send a batch again after a failure and get no duplicate rows. When you omit it, Oodle makes an id
input, output, expected	Any JSON
scores	Numbers from 0 to 1, by score name. A null score is ignored
error	A string or any JSON. It marks the row as failed
metrics	Numbers such as start and end (Unix seconds), prompt_tokens, completion_tokens, tokens and estimated_cost
metadata, tags	Your own data
traceId, spanId	Optional. Link the row to a trace that your code sent. Without a traceId, the row gets a trace of its own
datasetItemId	Optional. Link the row to a dataset item

Link rows to your own traces

When your code sends OpenTelemetry traces while it runs a case, put the trace id in the row's traceId. The row then opens that trace, with all the steps of your agent. Tag those traces with the project and the experiment so that they stay in the project:

export OTEL_RESOURCE_ATTRIBUTES="oodle.project=<PROJECT_NAME>,oodle.experiment.id=<EXPERIMENT_ID>"

Or send the X-Oodle-Project and X-Oodle-Experiment-Id headers with the OTLP export. See Send traces to a project.

Add feedback

Add scores or a comment to rows after you wrote them, for example after a human review:

curl -X POST "$OODLE_API/experiments/<EXPERIMENT_ID>/feedback" \
  -H "X-API-KEY: $OODLE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "feedback": [
      {"id": "case-0001", "scores": {"human_rating": 0.5}, "comment": "Correct, but the path is not exact.", "source": "ANNOTATION"}
    ]
  }'

id is your row id or the id that the insert returned. source is API (the default), ANNOTATION or EXTERNAL. The reply is {"rowIds": [...], "missing": [...]}, where missing lists the ids that matched no row.

Read the summary

curl "$OODLE_API/experiments/<EXPERIMENT_ID>/summary" \
  -H "X-API-KEY: $OODLE_API_KEY"

The summary gives, for each score and each metric, the average and the count. When there is a base experiment, it also gives the base average, the difference, and the number of rows that improved or regressed. Rows of the two experiments match on their input.

Oodle selects the base experiment in this order:

1. The compareTo=<experiment id> query parameter.
2. The baseExperimentId of the experiment.
3. The newest earlier experiment of the same project on the same git branch, then the newest earlier experiment of the project.

Compare experiments

curl "$OODLE_API/experiment-comparisons?experimentIds=<BASE_ID>,<NEW_ID>" \
  -H "X-API-KEY: $OODLE_API_KEY"

The reply lists the rows of two to five experiments, matched by input, with each row's scores in each experiment. It also has a summary of each experiment compared with the first one in the list. Use page and limit to read more rows.

Bring your own eval code

You can keep your eval code as it is, and send only the results to Oodle. This example runs a dataset through your agent, scores each case in your code, and uploads the rows. It uses the requests package.

import os
import subprocess
import sys

import requests

API = os.environ["OODLE_API"]
HEADERS = {"X-API-KEY": os.environ["OODLE_API_KEY"]}


def git(*args):
    return subprocess.check_output(["git", *args], text=True).strip()


def post(path, body):
    resp = requests.post(f"{API}/{path}", json=body, headers=HEADERS, timeout=60)
    resp.raise_for_status()
    return resp.json()


cases = [
    {"id": "case-0001", "input": "How do I reset my password?",
     "expected": "Settings > Security > Reset password"},
]

experiment = post("experiments", {
    "name": f"support-agent-{git('rev-parse', '--short', 'HEAD')}",
    "ensureNew": True,
    "parameters": {"model": "gpt-4.1"},
    "repoInfo": {"commit": git("rev-parse", "HEAD"),
                 "branch": git("rev-parse", "--abbrev-ref", "HEAD")},
})

rows = []
for case in cases:
    output = my_agent(case["input"])          # your code
    correct = 1.0 if case["expected"].lower() in output.lower() else 0.0
    rows.append({"id": case["id"], "input": case["input"], "output": output,
                 "expected": case["expected"], "scores": {"correctness": correct}})

# Send in batches. A batch that is sent again updates its rows.
for start in range(0, len(rows), 100):
    post(f"experiments/{experiment['id']}/rows", {"rows": rows[start:start + 100]})

summary = requests.get(f"{API}/experiments/{experiment['id']}/summary",
                       headers=HEADERS, timeout=60).json()
for score in summary["scores"]:
    print(f"{score['name']}: {score['average']:.2f} (diff {score.get('diff')})")

Run it in GitHub Actions

name: evals
on: pull_request

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install requests
      - run: python evals/run_evals.py | tee eval-summary.txt
        env:
          OODLE_API: https://<API_DOMAIN>/v1/api/instance/<OODLE_INSTANCE>/langfuse/api/public
          OODLE_API_KEY: ${{ secrets.OODLE_API_KEY }}
      - run: gh pr comment ${{ github.event.pull_request.number }} --body-file eval-summary.txt
        env:
          GH_TOKEN: ${{ github.token }}

Put a link to the experiment in the summary that your script prints, so that reviewers can open it from the pull request: https://<OODLE_UI>/genai/experiments/<EXPERIMENT_ID>. The page shows the scores against the base experiment and every row with its trace.

Upload failures

Oodle can answer an upload with a 5xx status for a short time. Because a row with the same id is updated and not added again, send the batch again with a backoff. Do not let an upload failure fail the eval run: keep the scores that your code computed, and decide pass or fail from them.