Webhook Evaluators

A webhook evaluator sends each span to an endpoint that you host, and stores the score that the endpoint returns. Use it when your scorer is already a service, when it needs your own data or models, or when it is in a language other than Python.

A webhook evaluator works like the other evaluators: a rule with filters and sampling scores live spans, and an experiment can select it to score its rows. It calls no LLM connection.

Set it up

1. Create the webhook: Agent Observability → Settings → Bring Your Own Agent (ap1, us1). The webhook holds the URL, the authentication, the headers, the timeout, and the output path and regex that read the score from the reply. See Bring Your Own Agent for these fields.
2. Create a template: Evaluators → Library → New template, then select Webhook. Select the webhook, and optionally write the request body.
3. Click Send test to send a sample span to your endpoint. The result shows the request, the reply and the score that Oodle read.
4. Create an evaluator (a rule) from the template, with filters and a sampling rate, or select it in an experiment.

The request

Oodle sends a POST with a JSON body. When you do not write a body, Oodle sends all the fields:

{
  "input": {{input}},
  "output": {{output}},
  "expected_output": {{expected_output}},
  "metadata": {{metadata}},
  "trace_id": {{trace_id}},
  "span_id": {{span_id}},
  "span_name": {{span_name}},
  "tool": {{tool}}
}

Write your own body to send only some fields, or to use the field names that your service reads. Type {{ in the editor to see the placeholders.

Placeholder	Value
{{input}}	The input of the span
{{output}}	The output of the span
{{expected_output}}	The expected output of the dataset item, in an experiment. null on a live span
{{metadata}}	The span attributes, other than the input, output and tool payloads
{{trace_id}}, {{span_id}}, {{span_name}}	The span's ids and name
{{tool.name}}, {{tool.arguments}}, {{tool.result}}	The tool call, on a tool span

Each placeholder becomes a JSON value, so do not put quotes around it. A string that holds a JSON object or array is sent as that JSON. A placeholder with no value sends null. The body must be valid JSON after Oodle fills it.

The webhook's headers go with each request. Use them for the authentication of your endpoint.

The reply

Reply with a 2xx status and one of these bodies. When the webhook has an output path or a regex, Oodle applies them first.

Reply	Score
A number from 0 to 1, for example 0.75	A numeric score
true or false	A boolean score: 1 or 0
{"score": 0.4, "label": "partial", "reason": "missed a step"}	The score, with the label and the reason. score can be a number from 0 to 1 or a boolean. label, reason and metadata (an object) are optional

The score has the name of the template. The reason appears as the comment of the score.

Failures

These count as a failure, not as a score:

• A status other than 2xx, a timeout, or a failed connection, after the retries.
• A reply that is empty, or that is not one of the forms above, for example a number above 1, a text, or an object without score.
• A reply with nothing at the output path, or that does not match the regex.

Oodle never writes a failure as a score of 0, because a 0 says that the answer was bad. On a live span, the failure shows in the evaluator's execution logs. In an experiment, the row shows the error.

Timeouts, retries and concurrency

Setting	Behavior
Timeout	The webhook's timeout, up to 30 seconds for each call
Retries	Oodle tries a call up to 3 times when it gets a 5xx or 429 status, a timeout, or a failed connection. The wait between tries doubles, from 1 second. A 4xx status other than 429 is not tried again
Concurrency	Oodle limits the number of calls that are in progress to one webhook, for all the rules and experiments that use it

Make your endpoint idempotent: a retry can send the same span again.

Network

The endpoint must be a public address that Oodle can reach from the internet. Use HTTPS, because the request carries your span data and your headers. Oodle refuses loopback, private and cluster-local addresses, also when a host name or a redirect resolves to one. To score with a service on your laptop during development, expose it with a tunnel such as ngrok.

To allow only Oodle to call your endpoint, put a secret in a webhook header and check it in your service. Oodle stores the header values encrypted.