Experiment Webhooks

An LLM connection tests one model call. A webhook tests your whole agent: Oodle posts each item of a dataset to an endpoint you host, your workflow does whatever it does, and the reply is scored with the same evaluators and output comparers a model's would be. Each result row links to the trace your agent emitted while it handled that item, and the evaluators can read that trace too.

Navigate to Agent Observability → Settings → Experiment Webhooks (ap1, us1), or pick Create one on the Run Experiment form.

note

Only Admin users can create and manage webhooks. Any user who can run an experiment can pick one. Header values are never shown again after they are saved: the editor shows them masked and keeps them unless you type a new one.

How a run works

1. Oodle reads the dataset and, for each item, renders the webhook's request body from the item.
2. It sends one POST per item to the URL, with the webhook's headers plus Content-Type, traceparent and X-Oodle-Experiment. Items run in parallel.
3. Any 2xx reply is accepted. The output path picks the output out of the reply; that text is stored on the run item.
4. If an evaluator will read it, Oodle waits for the agent's trace to arrive (up to 30 seconds) and turns it into a trajectory.
5. The evaluators and output comparers score the output, and each score is written to the run item and to the instance as a metric.

Parameters

Parameter	API field	Required	Description
Name	name	yes	Unique per instance. The Run Experiment form picks a webhook by name
Description	description	no	What the endpoint runs
URL	url	yes	Where each item is posted. Absolute http(s), reachable from the internet; loopback, private and cluster-local hosts are refused. For a local agent, expose it with a tunnel such as ngrok
Authentication	part of headers	no	None, a bearer token (Authorization: Bearer …) or basic auth. A view of the Authorization header
Headers	headers	no	Any other headers your endpoint needs, as a name to value map. Stored encrypted; the API returns only the names, and a write that sends ******** for a name keeps its stored value
Timeout	timeoutSeconds	no	How long one item may take, 1 to 600 seconds. Default 60. A test from the form is capped at 120
Request body	requestTemplate	no	JSON text with {{path}} placeholders. Default {{input}}: the item's input is the whole body
Output path	outputPath	no	Where the output is in the reply. Empty stores the whole reply

The headers traceparent, tracestate, X-Oodle-Experiment, Host and Content-Length are Oodle's and cannot be set.

Request body

A placeholder is replaced by the JSON of the value it names, so quoting and escaping are handled for you. In the form, type {{ to see the placeholders; the list includes the fields of the sample item's input.

Placeholder	Reads
{{input}}	The item's whole input
{{input.<field>}}	One field of an object input, such as {{input.question}}. Nest with dots and [n]: {{input.messages[0].content}}
{{metadata}}, {{metadata.<field>}}	The item's metadata
{{id}}	The dataset item id
{{run.id}}, {{run.name}}	The run's id and name (a number). test on a Send test
{{dataset.id}}, {{dataset.name}}	The dataset the run reads

For an endpoint that takes {"query": "...", "user": "eval"}:

{"query": {{input.question}}, "user": "eval"}

A placeholder whose path finds nothing renders as null. The expected output is never available to the template: it is ground truth for the evaluators only.

Output path

A dot path over the reply, with [n] for an array index: answer, result.text, choices[0].message.content. A string value is stored as is; anything else is stored as its JSON. A reply with nothing at the path fails the item.

What counts as a failure

A non-2xx status, a timeout, a reply larger than 4 MB, or a reply with nothing at the output path fails that item, and the reason is shown on the item's row. A 429, 502, 503 or 504 is retried once, with a fresh trace id. A timeout is not retried.

Testing a webhook

The editor shows the body it will send for a sample item as you type. Opened from a dataset's Run Experiment form, the sample is one of that dataset's items; otherwise it is any input you type. Send test posts that body and shows three things: the reply, the output the path picked out, and whether a trace with the request's trace id has arrived in Oodle.

The trace check keeps looking for a minute after the reply. Spans usually arrive a few seconds behind it, so "no trace yet" beside a reply is not a failure until the minute is up.

Linking your agent's trace to each result

Every request carries a W3C traceparent header holding a fresh trace id. A service with OpenTelemetry HTTP instrumentation continues that trace, so every span your agent emits while handling the item lands in Oodle under the same id and the result row opens it: the LLM calls, the tool calls, and what each one did.

Nothing else is needed. If your framework is already sending traces to Oodle (see the Agent Observability integrations), the server instrumentation that reads the incoming header is usually part of the same setup.

Telling experiment traffic apart

Every request also carries X-Oodle-Experiment: <run id>, so your agent can tell an experiment from production traffic. Two uses:

• Tag the traces. Capture the header as a span attribute and the traces page can filter on it. With OpenTelemetry's server instrumentation that is one setting, no code:

  # Python (FastAPI, Flask, Django, ASGI/WSGI)
  OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST=x-oodle-experiment
  # Java agent
  -Dotel.instrumentation.http.server.capture-request-headers=x-oodle-experiment

  The attribute is http.request.header.x_oodle_experiment on the server span.

• Keep experiments out of production numbers. Read the header and set deployment.environment (or whatever your agent uses for its environment) to something like experiment on the trace, so dashboards, insights and live evaluators filtered to production do not count it.

Evaluating the trajectory, not only the answer

An output comparer or LLM judge sees the item's input, the expected output and the output. On a webhook run it can also see how the agent got there: the steps of the trace, in order, with each tool call's name, arguments and result, each model call, its duration and whether it failed.

• In a judge or comparer prompt, use {{trajectory}}. It is filled with numbered lines such as:

  1. [agent] invoke_agent 2000ms
  2. [tool] lookup_order 300ms ERROR: timeout
     in: {"id": 42}
  3. [llm] ChatOpenAI.chat (gpt-4o) 1500ms

• In a code evaluator, ctx.trace.spans is the same list as dicts (kind, name, tool_name, model, input, output, duration_ms, error, error_message), and ctx.trace.tools_called() lists the tool names in order:

  def evaluate(ctx):
      called = ctx.trace.tools_called()
      return EvaluationResult(scores=[
          Score(name="looked_up_order", value=1.0 if "lookup_order" in called else 0.0),
          Score(name="steps", value=len(ctx.trace.spans)),
      ])

Oodle waits up to 30 seconds after the reply for the trace to finish arriving, and only when an evaluator will read it. Long arguments and results are cut at 2,000 characters; the whole trace is on the run item's row. On an LLM connection run, and on a webhook run whose trace never arrived, the trajectory is empty.

Running an experiment against a webhook

On the Run Experiment form, choose Your own agent and pick the webhook. No prompt or model is asked for: the endpoint owns both.

A run against a webhook has no generation connection, so the LLM judges and output comparers you pick need one of their own. An evaluator that names its own connection uses it; the Evaluator LLM connection field covers any that do not. Code evaluators run no model and need none.

Results read the same as any other run. Each row shows the output, its scores, and the trace, with the span count, latency, tokens and cost the trace carried.

API

Under /v1/api/instance/{instance}/langfuse/api/public/:

Method and path	Purpose
GET webhooks	List webhooks (header names only)
POST webhooks	Create one. Body: the parameters above, headers as a map
GET webhooks/{id}	One webhook
PUT webhooks/{id}	Update. Omitted fields keep their value; omitted headers keeps the stored ones, and a header sent as ******** keeps that one's stored value
DELETE webhooks/{id}	Delete
POST webhook-tests	Send one request the way a run would. Body: webhookId or the parameters inline, plus input or datasetItemId. Returns the request sent (credentials masked), the reply, the extracted output and the trace id
POST jobs with type: "llm-experiment" and config.webhookId	Queue a run. config.evalConnectionId for the judges

CLI

The Oodle CLI manages webhooks under oodle genai webhooks and runs experiments against one with --webhook-id:

oodle genai webhooks create --name "Support agent" \
  --url https://agent.example.com/run \
  --header "Authorization=Bearer $AGENT_TOKEN" \
  --request-template '{"query": {{input.question}}, "run": {{run.name}}}' \
  --output-path answer --timeout 120

oodle genai webhooks test "$WH" --input '{"question": "Is checkout slow?"}'

oodle genai experiments run --dataset-id "$DS" --webhook-id "$WH" \
  --output-comparer-id oodle-managed-output-match-v1 \
  --eval-connection-id "$CONN"

list, get, update -f <file> and delete complete the set. Header values are encrypted at rest and never returned; get shows their names.

Terraform

The Oodle Terraform provider has an oodle_genai_webhook resource, so a webhook can live beside the service it points at:

resource "oodle_genai_webhook" "support_agent" {
  name        = "support-agent-staging"
  url         = "https://agent-staging.example.com/run"

  headers = {
    Authorization = "Bearer ${var.agent_token}"
  }

  request_template = "{\"query\": {{input.question}}}"
  output_path      = "answer"
  timeout_seconds  = 120
}

Header values are stored encrypted and the API returns only their names, so keep them in a variable or a secret store.

MCP

Through the Oodle MCP server, an AI assistant can manage webhooks with manage_genai_webhooks (list, get, create, update, delete; header values are never returned, and an update that sends ******** for a header keeps its stored value) and queue a run against one with manage_genai_experiments:

{"action": "run", "datasetId": "<dataset id>", "webhookId": "<webhook id>",
 "outputComparerIds": ["oodle-managed-output-match-v1"],
 "evalConnectionId": "<connection id>"}

A credential still has to be typed somewhere to be stored, so an assistant should create the webhook without one and point you at the editor to add it, rather than ask for it in the conversation.