TraceQL

Oodle supports TraceQL, the trace query language of Grafana Tempo. Use TraceQL to:

• Search: find traces that have spans that match a filter.
• Compute metrics: make time series from the matching spans, for example a rate, a count, an average or a quantile, grouped by any attribute.

TraceQL metrics read the stored spans, so they answer questions about the past and use any span attribute. For alerts, use span metrics with a PromQL monitor. See Alerting on Logs and Traces.

Run a Query

Where	How
Grafana Explore	In the Oodle UI, go to Dashboards, then Explore. Select the oodle-tempo data source and the TraceQL query type.
HTTP API	Call the TraceQL endpoints with an API key. See HTTP API.
Oodle CLI	Run oodle traces traceql search, metrics, tags or tag-values. See CLI.
Oodle MCP server	AI agents call the query_traceql tool. See MCP.

Search Queries

A search query is a spanset filter in braces. A trace matches when one or more of its spans match the filter.

{ resource.service.name = "checkout" && status = error }

Intrinsics

Intrinsic	Type	Example
name	string	{ name = "GET /orders" }
status	ok, error, unset	{ status = error }
statusMessage	string	{ statusMessage =~ ".*timeout.*" }
duration	duration	{ duration > 2s }
kind	server, client, producer, consumer, internal, unspecified	{ kind = server }
traceID	string	{ traceID = "4bf92f3577b34da6a3ce929d0e0e4736" }
spanID	string	{ spanID = "00f067aa0ba902b7" }
parentID	string	{ parentID = "00f067aa0ba902b7" }

You can also write the scope-qualified form, for example span:name, span:duration or span:status.

Durations use the units ns, us, ms, s, m and h, for example 500ms or 2s.

Attributes

Form	Looks in	Example
span.<name>	Span attributes	{ span.http.status_code >= 500 }
resource.<name>	Resource attributes	{ resource.service.name = "api" }
.<name>	Span attributes and resource attributes	{ .http.method = "POST" }
instrumentation.<name>	Instrumentation scope attributes	{ instrumentation.team = "payments" }
event.<name>	Span event attributes	{ event.exception.type = "TimeoutError" }

Put a name that has spaces or special characters in double quotes: { span."my attr" = "v" }.

Operators

Operator	Meaning
=, !=	Equal, not equal
>, >=, <, <=	Numeric and duration comparison
=~, !~	Regular expression match, no match
= nil, != nil	Attribute is absent, attribute is present
&&, ||, !	AND, OR and NOT inside one filter
{ A } || { B }	Traces with a span that matches A or a span that matches B

Search Examples

Errors in one service:

{ resource.service.name = "checkout" && status = error }

Slow server spans:

{ kind = server && duration > 2s }

HTTP 5xx responses on one route:

{ span.http.route = "/api/orders" && span.http.status_code >= 500 }

Spans that do not come from health checks:

{ resource.service.name = "api" && !(name =~ ".*health.*") }

Spans that have a user attribute:

{ span.user.id != nil }

GenAI tool calls that took more than 10 s:

{ name =~ "execute_tool.*" && duration > 10s }

Metrics Queries

A metrics query is a spanset filter, a pipe (|) and a metrics function. Add by (...) to get one series for each value of one or more intrinsics or attributes.

Function	Result
rate()	Matching spans per second.
count_over_time()	Matching spans in each step.
avg_over_time(<field>)	Average of a numeric field in each step.
min_over_time(<field>)	Minimum of a numeric field in each step.
max_over_time(<field>)	Maximum of a numeric field in each step.
sum_over_time(<field>)	Sum of a numeric field in each step.
quantile_over_time(<field>, <q>, ...)	One or more quantiles of a numeric field, for example 0.5, 0.95, 0.99.
histogram_over_time(<field>)	Distribution of a numeric field in buckets.

<field> is duration or a numeric attribute, for example span.gen_ai.usage.output_tokens.

Metrics Examples

Error rate for each route of a service:

{ resource.service.name = "api" && status = error } | rate() by (span.http.route)

Number of GenAI tool calls over 10 s for each tool:

{ name =~ "execute_tool.*" && duration > 10s } | count_over_time() by (span.gen_ai.tool.name)

p95 and p99 duration of server spans for each service:

{ kind = server } | quantile_over_time(duration, 0.95, 0.99) by (resource.service.name)

Average output tokens for each model:

{ span.gen_ai.usage.output_tokens > 0 } | avg_over_time(span.gen_ai.usage.output_tokens) by (span.gen_ai.request.model)

Spans for each service:

{} | count_over_time() by (resource.service.name)

Write These Queries in a Supported Form

Some TraceQL forms compare more than one span, or compare a trace as a whole. Oodle evaluates each filter on one span at a time. Use the form in the right column:

Query form	Use this form
{ A } && { B } (two different spans)	{ A && B } when one span has both conditions. Else run one search for each filter.
Scalar filter, such as { A } | count() > 2	{ A } | count_over_time() by (...), then read the counts.
Structural operators >>, <<, ~	Filter on the span itself, for example with name, kind or resource.service.name.
parent. and link. attributes	Filter on the attributes of the span itself.
rootName, traceDuration	name and duration of a span. For root spans and request counts, use oodle_trace_metrics with is_root_span="true".
Bare attribute, such as { .user.id }	{ .user.id != nil }
Attribute compared to attribute, such as { .a = .b }	Compare each attribute to a literal value.

HTTP API

Base path: https://<OODLE_INSTANCE>.api.oodle.ai/v1/api/instance/<OODLE_INSTANCE>/traces/traceql. Send the headers X-OODLE-INSTANCE: <OODLE_INSTANCE> and X-API-KEY: <OODLE_API_KEY>. Get both values from Settings > API Keys.

Method and path	Parameters	Returns
GET /search	q (TraceQL filter), start, end, limit	Matching traces
GET /metrics/query_range	q (TraceQL metrics query), start, end, step	Time series
GET /tags	q (optional filter)	Attribute names
GET /tags/{tagName}/values	q (optional filter)	Values of one attribute

• start and end are Unix times in seconds. When you do not set them, the query uses the most recent period.
• step is a duration such as 30s, 5m or 1h, or a number of seconds. When you do not set it, Oodle picks a step that gives about 100 points.
• limit is the maximum number of traces. The default is 20 and the maximum is 1000.
• The responses use the Grafana Tempo response format.

Search for errors in the last hour:

END=$(date +%s); START=$((END - 3600))
curl -G "https://<OODLE_INSTANCE>.api.oodle.ai/v1/api/instance/<OODLE_INSTANCE>/traces/traceql/search" \
  -H "X-OODLE-INSTANCE: <OODLE_INSTANCE>" \
  -H "X-API-KEY: <OODLE_API_KEY>" \
  --data-urlencode 'q={ resource.service.name = "checkout" && status = error }' \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END" \
  --data-urlencode "limit=50"

Error rate for each route over the last 6 hours, in 5-minute steps:

END=$(date +%s); START=$((END - 21600))
curl -G "https://<OODLE_INSTANCE>.api.oodle.ai/v1/api/instance/<OODLE_INSTANCE>/traces/traceql/metrics/query_range" \
  -H "X-OODLE-INSTANCE: <OODLE_INSTANCE>" \
  -H "X-API-KEY: <OODLE_API_KEY>" \
  --data-urlencode 'q={ resource.service.name = "api" && status = error } | rate() by (span.http.route)' \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END" \
  --data-urlencode "step=5m"

CLI

The Oodle CLI has a traces traceql command group. --start and --end take relative times such as -6h.

# Search
oodle traces traceql search '{ resource.service.name="api" && status=error }'
oodle traces traceql search '{ span.http.status_code >= 500 }' --start -6h --limit 50

# Metrics
oodle traces traceql metrics '{ resource.service.name="api" && status=error } | rate() by (span.http.route)'
oodle traces traceql metrics '{ name=~"execute_tool.*" && duration > 10s } | count_over_time() by (span.gen_ai.tool.name)' --start -24h
oodle traces traceql metrics '{ resource.service.name="api" } | quantile_over_time(duration, 0.95)' --step 5m

# Attribute names and values
oodle traces traceql tags
oodle traces traceql tag-values resource.service.name
oodle traces traceql tag-values span.http.route --query '{ resource.service.name="api" }'