Migrate from Braintrust

Oodle serves a Braintrust-compatible API. Your code that uses the Braintrust SDK (Python or TypeScript) sends its experiments, datasets, scores and feedback to Oodle when you change some environment variables. You do not change the code.

This page is for teams that:

• Run evals in their own code with braintrust.init(), Eval() or init_dataset(), on a laptop, a server or in CI.
• Send production traces with OpenTelemetry. See OpenTelemetry.
• Want to keep their history: the import script copies projects, datasets and experiments from Braintrust to Oodle.

Set the environment variables

export BRAINTRUST_APP_URL="https://<API_DOMAIN>/v1/api/instance/<OODLE_INSTANCE>/braintrust"
export BRAINTRUST_API_URL="https://<API_DOMAIN>/v1/api/instance/<OODLE_INSTANCE>/braintrust"
export BRAINTRUST_API_KEY="<OODLE_API_KEY>"
# Optional: the links that the SDK prints open in Oodle.
export BRAINTRUST_APP_PUBLIC_URL="https://<OODLE_UI>"

Variable	Value
BRAINTRUST_APP_URL	The Braintrust-compatible base URL of your Oodle instance
BRAINTRUST_API_URL	The same URL
BRAINTRUST_API_KEY	An Oodle API key (Settings → API Keys)
BRAINTRUST_APP_PUBLIC_URL	Optional. The address of the Oodle UI that you sign in to, for example https://us1.oodle.ai. The SDK uses it for the experiment links that it prints. When it is not set, the links go to the API, which sends them on to the Oodle UI

<API_DOMAIN> and <OODLE_INSTANCE> are on the Braintrust tile in Integrations, filled in for your instance.

Run your code

The code does not change. These examples run with the variables above.

Python

import braintrust
from braintrust import Eval, init, init_dataset

PROJECT = "support-agent"

# An experiment that your code fills row by row.
experiment = init(project=PROJECT, experiment="prompt-v2")
for case in cases:
    output = my_agent(case["input"])
    experiment.log(
        input=case["input"],
        output=output,
        expected=case["expected"],
        scores={"exact": 1.0 if output == case["expected"] else 0.0},
        metadata={"category": case["category"]},
    )
print(experiment.summarize())

# A dataset.
dataset = init_dataset(project=PROJECT, name="golden")
dataset.insert(input={"q": "How do I reset my password?"},
               expected="Settings > Security > Reset password", id="item-1")
dataset.flush()


# Eval() with scorers in your code, over the dataset.
def exact(input, output, expected):
    return 1.0 if output == expected else 0.0


Eval(
    PROJECT,
    data=init_dataset(project=PROJECT, name="golden"),
    task=my_agent,
    scores=[exact],
    experiment_name="eval-run",
)

TypeScript

import { Eval, init, initDataset } from "braintrust";

const PROJECT = "support-agent";

// An experiment that your code fills row by row.
const experiment = init({ project: PROJECT, experiment: "prompt-v2" });
for (const c of cases) {
  const output = await myAgent(c.input);
  experiment.log({
    input: c.input,
    output,
    expected: c.expected,
    scores: { exact: output === c.expected ? 1 : 0 },
  });
}
console.log(await experiment.summarize());

// A dataset.
const dataset = initDataset({ project: PROJECT, dataset: "golden" });
dataset.insert({
  input: { q: "How do I reset my password?" },
  expected: "Settings > Security > Reset password",
  id: "item-1",
});
await dataset.flush();

// Eval() with scorers in your code, over the dataset.
await Eval(PROJECT, {
  data: initDataset({ project: PROJECT, dataset: "golden" }),
  task: (input) => myAgent(input),
  scores: [
    ({ output, expected }) => ({ name: "exact", score: output === expected ? 1 : 0 }),
  ],
  experimentName: "eval-run",
});

How Braintrust objects map to Oodle

Braintrust	Oodle
Project	Project. The SDK makes the project when it does not exist
Experiment	Experiment in the project. See the Experiments API
Experiment row	Experiment row, linked to its GenAI trace
Dataset	Dataset in the project
Production traces, sent with OpenTelemetry	GenAI traces of the project. The traces carry oodle.project
Scores, also the scores of scorer spans in Eval()	Scores on the row and on its trace. A scorer span's score rolls up to its row
Feedback (log_feedback) on experiment and dataset rows	Scores, and comments on the row
Span types (llm, task, tool, score, function, eval)	GenAI span kinds and names
metrics (prompt_tokens, completion_tokens, tokens)	Token usage, and cost from your model pricing
metadata, tags	Metadata and tags

The traces of an experiment have the experiment id in oodle.experiment.id. They are in the project, so they stay out of the views of the default project.

What the Braintrust-compatible API serves

SDK call or feature	Served
Login with an API key	BRAINTRUST_API_KEY is your Oodle API key
init(), log(), summarize()	Experiments, rows, and summaries compared with the base experiment
Eval() with scorers in your code	The task, the scorers and the rows, with scorer spans
init_dataset(): insert(), iteration, fetch()	Datasets and their rows
log_feedback()	Scores and comments on experiment rows and dataset rows
Base experiments and comparisons	base_experiment and the comparison in summarize()
REST API under /v1/project, /v1/experiment, /v1/dataset	Create, list, get, update, delete, insert, fetch, feedback and summarize
OpenTelemetry export with x-bt-parent	Traces of the project or the experiment. See OpenTelemetry

Some Braintrust features have an Oodle equivalent that you set up in Oodle:

To do this	Use
Score with code or an LLM judge that runs on the server	Evaluators, or scorers in your Eval() code
Score with your own service	A webhook evaluator
Query logs and experiments	The traces list and its filters, or fetch() for the rows of one object
Run an eval from the command line	Run the file that holds your Eval() call with python or node
Control who can read and change data	Oodle user roles
Score production traces automatically	Evaluators with a filter on the project
Send images and files	Put the content in the row's input or output

OpenTelemetry

Production traces go to Oodle with OpenTelemetry. If you send OpenTelemetry traces to Braintrust, change the endpoint and the key. Keep the x-bt-parent header:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<API_DOMAIN>/v1/api/instance/<OODLE_INSTANCE>/braintrust/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <OODLE_API_KEY>,x-bt-parent=project_name:support-agent"
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip

The exporter adds /v1/traces to the endpoint. x-bt-parent takes one of these forms:

Value	The traces go to
project_name:<name>	The project with this name. Oodle makes it when it does not exist
project_id:<id>	The project with this id
experiment_id:<id>	The experiment, and the project of the experiment

Oodle reads the braintrust.* span attributes (input, output, expected, metadata, metrics, scores, tags and span attributes) and shows them as GenAI fields. Keep compression on: GenAI spans hold whole prompts and answers.

You can also send the traces to the Oodle OTLP endpoint and tag them with oodle.project. See Send traces to a project.

Copy your history from Braintrust

The import script copies projects, datasets and experiments, with their rows and scores, from Braintrust to Oodle. It needs Python 3.9 or later and only the standard library.

Download braintrust_import.py, then run it:

python braintrust_import.py \
  --source-api-key "$BRAINTRUST_API_KEY" \
  --dest-url "https://<API_DOMAIN>/v1/api/instance/<OODLE_INSTANCE>/braintrust" \
  --dest-api-key "$OODLE_API_KEY" \
  --project support-agent --project billing-agent

Flag	Description
--source-api-key	Your Braintrust API key. Required
--dest-url	The Braintrust-compatible URL of your Oodle instance. Required
--dest-api-key	Your Oodle API key. Required
--project NAME	A project to copy. Use it again for more projects. Without it, the script copies all projects
--source-url	The Braintrust API URL. The default is https://api.braintrust.dev
--dry-run	Read from Braintrust and count the rows. Write nothing to Oodle
--quiet	Print less

You can run the script again. It finds projects, datasets and experiments by name, and the rows keep their Braintrust ids, so a second run updates the rows of the first run and makes no copies. Start with --dry-run to see the counts.

Production traces come to Oodle with OpenTelemetry, so the script copies experiments and datasets.

Failures and retries

When Oodle cannot reach its trace store for a short time, the API answers 503. The SDK sends the request again. Writes are idempotent by row id: a row that is sent two times is stored one time.

An upload failure does not make your eval results wrong: your code computed the scores. In CI, decide pass or fail from your scores, and catch a failure of the upload so that it does not fail the job:

Python

import sys

import braintrust

experiment = braintrust.init(project="support-agent", experiment="ci-run")
scores = []
for case in cases:
    output = my_agent(case["input"])
    score = 1.0 if output == case["expected"] else 0.0
    scores.append(score)
    experiment.log(input=case["input"], output=output,
                   expected=case["expected"], scores={"exact": score})

try:
    print(experiment.summarize())
except Exception as err:  # The report failed, not the eval.
    print(f"warning: upload to Oodle failed: {err}", file=sys.stderr)

average = sum(scores) / len(scores)
sys.exit(0 if average >= 0.9 else 1)

TypeScript

import { init } from "braintrust";

const experiment = init({ project: "support-agent", experiment: "ci-run" });
const scores: number[] = [];
for (const c of cases) {
  const output = await myAgent(c.input);
  const score = output === c.expected ? 1 : 0;
  scores.push(score);
  experiment.log({ input: c.input, output, expected: c.expected, scores: { exact: score } });
}

try {
  console.log(await experiment.summarize());
} catch (err) {
  // The report failed, not the eval.
  console.warn(`warning: upload to Oodle failed: ${err}`);
}

const average = scores.reduce((a, b) => a + b, 0) / scores.length;
process.exit(average >= 0.9 ? 0 : 1);

With Eval(), put braintrust.flush() (Python) or await flush() (TypeScript) in the same try block after the Eval() call.

Next steps

• Projects: keep experiment traffic apart from production.
• Experiments API: the Oodle API for experiments, for code that does not use the Braintrust SDK.
• Evaluators: score production traffic in Oodle.
• Coming from another tool: move Braintrust scorers to Oodle code evaluators.