Skip to main content

Agent Observability

Oodle's Agent Observability helps engineering teams track how large language models are used across their applications - including prompts, responses, token consumption, and estimated costs.

How It Works​

Oodle ingests traces and logs that follow the OpenTelemetry Gen AI semantic conventions. When your application instruments LLM calls with these conventions, Oodle automatically:

  • Detects the model from gen_ai.request.model and gen_ai.response.model span attributes
  • Extracts input and output token counts from gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
  • Displays prompt and response content from gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions span attributes
  • Calculates estimated costs based on your configured model pricing

All of this is surfaced in the Agent Observability hub at Agent Observability → Overview (ap1, us1).

Exploring Traces​

The Agent Observability hub has several tabs for different views of your GenAI data.

Overview​

The default tab shows an embedded Grafana dashboard with key metrics: token usage, cost trends, trace counts, and model breakdowns over time.

Traces​

The main trace list with columns for timestamp, name, input, output, model, sentiment, turns, latency, and cost.

LLM Traces list showing model, input, output, latency, and cost for each trace

Search traces by Name, Input, Output, or Input/Output combined using the field dropdown next to the search box.

Click any trace to open the detail view (see Trace Detail below).

Agent Graph​

An interactive topology graph showing how agents, tools, and LLM calls connect in your application. Each node displays request count, error rate, and average latency. Click a node to see:

  • Decision Paths — outgoing connections split into sub-agents and tool calls with probability, error rate, and latency
  • Traces — filtered trace list for that agent
  • Insights — AI-generated recommendations specific to that agent (if available)

The Agent Graph has its own sidebar with environment and service filters, plus a node search and visibility toggle.

Observations​

A span-level table showing individual LLM calls rather than full traces. Columns include start time, span type, name, input, output, model, sentiment, log level, latency, and cost.

Sessions​

Groups traces by session.id into multi-turn conversations. Each row shows the session ID, trace count, turns, sentiment, total cost, duration, and last activity time. Click a session to open a detail drawer with stats and a cross-trace transcript.

tip

To enable sessions, set session.id as a span attribute in your instrumentation. Traces sharing the same session ID are grouped together automatically.

Users​

Aggregates usage by user.id — showing total events, turns, sentiment, token count, and cost per user. Click a user to filter the Traces tab to that user's activity.

tip

Set user.id as a span attribute to enable per-user tracking.

Insights​

Appears when Oodle has generated AI-powered recommendations for your GenAI application. Insights are categorized by type (Quality, Efficiency, Resource, Performance) and severity (High, Medium, Low). Use this tab to identify optimization opportunities and quality issues.

The sidebar on the left lets you filter data across most tabs:

FilterDescription
Time RangeControls the query window
Has ErrorShow only traces with errors
EnvironmentFilter by resource::env
ServiceFilter by resource::service.name
ModelFilter by gen_ai.request.model
Agent NameFilter by gen_ai.agent.name
Operation NameFilter by gen_ai.operation.name
User IDFilter by user.id
SentimentRange slider from −1 to 1
ScoreRange slider from 0 to 1
Score NameFilter by evaluator score name

You can also add arbitrary label filters using the filter bar above the tab content.

Trace Detail​

Clicking on a trace opens the detail view with tabs for Transcript, Trace, Logs, Scores, and Costs.

Transcript​

Displays the full conversation — system instructions, user prompts, and assistant responses — alongside the model name and token count.

Trace detail showing the Transcript tab with system, user, and assistant messages

Use the search bar to find text across all messages. Matches are highlighted and the view scrolls to each result.

For traces with multiple agents (identified by gen_ai.agent.name), a By Agent toggle groups the conversation by agent.

Trace​

Shows the span waterfall with the full request lifecycle, including nested agent invocations and LLM calls.

Trace span waterfall showing nested agent and LLM call hierarchy

By default, non-GenAI infrastructure spans are hidden. Toggle Show all spans to reveal them. GenAI-related spans (GENERATION, AGENT, TOOL) are expanded automatically while infrastructure spans start collapsed.

Each span has sub-tabs for Overview, Gen AI (model/token/cost summary), Transcript, and Errors.

Scores​

Shows all evaluator scores attached to the trace. Each row displays the score name, value, source (EVAL, ANNOTATION, or API), data type, model used, and comment.

Costs​

Summarizes total cost, tokens, and LLM calls broken down by model. Shows separate breakdowns for model costs and evaluator costs.

Costs tab showing total cost, tokens, and cost by model breakdown

Trace Actions​

The trace detail toolbar includes several actions:

ActionDescription
Analyze with AIOpens the AI sidebar assistant with the trace as context
Open in PlaygroundLoads the trace conversation into the Playground for iteration
Add to DatasetAdds the trace as a test case to a dataset
Open TraceOpens the trace in the standard trace lookup view

Getting Started​

1. Choose how data reaches Oodle​

There are two supported paths. Both send the same OpenTelemetry data, so pick the one that matches your environment.

Via an OpenTelemetry Collector. Your application exports to a collector you run, and the collector forwards to Oodle. Redaction, sampling and batching live in the collector, and every service uses one export path. Use this if you already run a collector.

Direct to Oodle. Your application exports to Oodle with no collector in between. There is nothing extra to operate, but the application holds the Oodle API key and there is no collector hop to redact content. Use the SDK's own options to limit what is captured.

Oodle's OTLP endpoint serves the standard OTLP signal paths (/v1/traces, /v1/logs, /v1/metrics) and requires the X-API-KEY and X-OODLE-INSTANCE headers. Requests without them are rejected with 401.

2. Install instrumentation​

Install the instrumentation for your LLM provider. Google Gemini, Pydantic AI and LiteLLM are Python only: there is no OpenTelemetry instrumentation for the @google/genai JavaScript SDK, Pydantic AI has no JavaScript port, and LiteLLM carries its OTel callback in the Python SDK only.

pip install langfuse openai \
  opentelemetry-exporter-otlp-proto-http \
  opentelemetry-sdk

See the OpenTelemetry GenAI instrumentation registry for a full list of supported providers (Anthropic, AWS Bedrock, and more).

3. Instrument your application​

The examples below export to a collector at http://otel-collector:4318. For the direct path, replace that endpoint with your Oodle OTLP endpoint and add the authentication headers, as shown in step 4.

JavaScript: initialization order matters

OpenTelemetry auto-instrumentation patches modules as they load. The SDK initialization must run before the LLM client is required. If you require the client first, the application works but emits no spans.

The Langfuse SDK is built on OpenTelemetry, so its spans can be sent to Oodle like any other instrumentation. Oodle resolves the langfuse.* attributes to the standard gen_ai.* attributes when it ingests them.

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import set_tracer_provider
from langfuse import Langfuse, get_client, observe


def setup_opentelemetry():
    resource = Resource.create({"service.name": "my-llm-app"})

    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter())
    )
    set_tracer_provider(tracer_provider)
    return tracer_provider


# Langfuse writes into the application tracer provider.
Langfuse(tracer_provider=setup_opentelemetry())

# The Langfuse OpenAI drop-in traces every call it makes.
from langfuse.openai import openai


@observe()
def chat(message: str) -> str:
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": message}],
    )
    return response.choices[0].message.content


chat("Hello!")

# Short-lived processes must flush before they exit.
get_client().flush()
caution

Attach Langfuse to the application tracer provider, as shown. If you register LangfuseSpanProcessor and an OTLP span processor in the same provider, every Langfuse span is exported twice, which double counts tokens and cost. See the Langfuse guide on using an existing OpenTelemetry setup.

note

LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY must be set even when the OTLP exporter owns transport. Without them the Langfuse client disables itself and emits nothing. Any value works in that case.

4. Set environment variables​

export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_and_event

The collector holds the Oodle credentials, so the application does not need them.

VariableDescription
OTEL_EXPORTER_OTLP_ENDPOINTOTLP endpoint: your collector, or the Oodle OTLP endpoint for the direct path
OTEL_EXPORTER_OTLP_HEADERSOodle authentication, direct path only
OTEL_SEMCONV_STABILITY_OPT_INSet to gen_ai_latest_experimental to enable GenAI semantic conventions (not needed for Pydantic AI). Required for LiteLLM: without it the span carries no gen_ai.operation.name, and the Agent Observability traces list, which filters on that attribute, shows nothing
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENTCaptures prompt/response content. Python reads a mode name, so use span_and_event; the JavaScript instrumentation parses a boolean, so use true there. Not needed for Pydantic AI, which captures content by default
caution

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT captures full prompt and response text. This may include sensitive data. Review your organization's data policies before you enable it in production. On the direct path there is no collector to redact it.

5. Configure the OpenTelemetry Collector​

Skip this step if you send data directly to Oodle.

Route traces and logs from your application to Oodle through an OpenTelemetry Collector. The collector exports to Oodle's OTLP endpoint, which serves the standard OTLP paths (/v1/traces, /v1/logs):

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  otlphttp/oodle:
    endpoint: "https://<OODLE_INSTANCE>-otlp.collector.oodle.ai"
    headers:
      "X-OODLE-INSTANCE": "<OODLE_INSTANCE>"
      "X-API-KEY": "<OODLE_API_KEY>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/oodle]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/oodle]

Replace <OODLE_INSTANCE> and <OODLE_API_KEY> with your credentials from Settings → API Keys in the Oodle UI (ap1, us1).

See the OpenTelemetry integration guide for full collector and SDK configuration options.

6. Verify in Oodle​

Navigate to GenAI > Traces (ap1, us1). Click on a trace to see the Transcript, Trace waterfall, and Costs breakdown.

If no spans arrive:

  • JavaScript, no spans at all: confirm the SDK initialization runs before the LLM client is required, and that openai is below version 7.
  • Python with OpenAI, ModuleNotFoundError: install httpx.
  • Langfuse, no spans: confirm LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set.
  • Langfuse spans counted twice: the application must not add its own OTLP span processor as well as LangfuseSpanProcessor.
  • 401 responses: check X-API-KEY and X-OODLE-INSTANCE.

Demo Applications​

The oodle-onboarding repository includes ready-to-run demos:

  • llmops-otel-demo — Google Gemini with OTel instrumentation
  • pydantic-ai-demo — Pydantic AI multi-agent example with tool use and structured outputs
  • traceloop-demo — Traceloop (OpenLLMetry) SDK auto-instrumenting Google Gemini
  • openlit-demo — OpenLIT with agents (LangGraph), guardrails, VectorDB (ChromaDB RAG), and MCP tool tracing
git clone https://github.com/oodle-ai/oodle-onboarding.git
cd oodle-onboarding/llmops-otel-demo
cp .env.example .env
# Edit .env with your OODLE_INSTANCE, OODLE_API_KEY, and GEMINI_API_KEY
make up
make test-chat

Each demo includes an OTel Collector and Docker Compose configuration — everything needed to see LLM traces in Oodle in minutes.

Cost Tracking​

Once model definitions are configured, Oodle shows cost data on every LLM trace. The cost is calculated as:

cost = (input_tokens / 1,000,000) × input_price
     + (output_tokens / 1,000,000) × output_price

For parent spans (agent invocations), Oodle aggregates token counts from child LLM spans and shows the total estimated cost. See the Model Definitions page for pricing configuration.

Support

If you need assistance or have any questions, please reach out to us through: