Skip to main content

Agent Observability

Oodle's Agent Observability helps engineering teams track how large language models are used across their applications - including prompts, responses, token consumption, and estimated costs.

How It Works​

Oodle ingests traces and logs that follow the OpenTelemetry Gen AI semantic conventions. When your application instruments LLM calls with these conventions, Oodle automatically:

  • Detects the model from gen_ai.request.model and gen_ai.response.model span attributes
  • Extracts input and output token counts from gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
  • Displays prompt and response content from gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions span attributes
  • Calculates estimated costs based on your configured model pricing

All of this is surfaced in the Agent Observability hub at Agent Observability → Overview (ap1, us1).

Exploring Traces​

The Agent Observability hub has several tabs for different views of your GenAI data.

Overview​

The default tab shows an embedded Grafana dashboard with key metrics: token usage, cost trends, trace counts, and model breakdowns over time.

Traces​

The main trace list with columns for timestamp, name, input, output, model, sentiment, turns, latency, and cost.

LLM Traces list showing model, input, output, latency, and cost for each trace

Search traces by Name, Input, Output, or Input/Output combined using the field dropdown next to the search box.

Click any trace to open the detail view (see Trace Detail below).

Agent Graph​

An interactive topology graph showing how agents, tools, and LLM calls connect in your application. Each node displays request count, error rate, and average latency. Click a node to see:

  • Decision Paths — outgoing connections split into sub-agents and tool calls with probability, error rate, and latency
  • Traces — filtered trace list for that agent
  • Insights — AI-generated recommendations specific to that agent (if available)

The Agent Graph has its own sidebar with environment and service filters, plus a node search and visibility toggle.

Observations​

A span-level table showing individual LLM calls rather than full traces. Columns include start time, span type, name, input, output, model, sentiment, log level, latency, and cost.

Sessions​

Groups traces by session.id into multi-turn conversations. Each row shows the session ID, trace count, turns, sentiment, total cost, duration, and last activity time. Click a session to open a detail drawer with stats and a cross-trace transcript.

tip

To enable sessions, set session.id as a span attribute in your instrumentation. Traces sharing the same session ID are grouped together automatically.

Users​

Aggregates usage by user.id — showing total events, turns, sentiment, token count, and cost per user. Click a user to filter the Traces tab to that user's activity.

tip

Set user.id as a span attribute to enable per-user tracking.

Insights​

Appears when Oodle has generated AI-powered recommendations for your GenAI application. Insights are categorized by type (Quality, Efficiency, Resource, Performance) and severity (High, Medium, Low). Use this tab to identify optimization opportunities and quality issues.

The sidebar on the left lets you filter data across most tabs:

FilterDescription
Time RangeControls the query window
Has ErrorShow only traces with errors
EnvironmentFilter by resource::env
ServiceFilter by resource::service.name
ModelFilter by gen_ai.request.model
Agent NameFilter by gen_ai.agent.name
Operation NameFilter by gen_ai.operation.name
User IDFilter by user.id
SentimentRange slider from −1 to 1
ScoreRange slider from 0 to 1
Score NameFilter by evaluator score name

You can also add arbitrary label filters using the filter bar above the tab content.

Trace Detail​

Clicking on a trace opens the detail view with tabs for Transcript, Trace, Logs, Scores, and Costs.

Transcript​

Displays the full conversation — system instructions, user prompts, and assistant responses — alongside the model name and token count.

Trace detail showing the Transcript tab with system, user, and assistant messages

Use the search bar to find text across all messages. Matches are highlighted and the view scrolls to each result.

For traces with multiple agents (identified by gen_ai.agent.name), a By Agent toggle groups the conversation by agent.

Trace​

Shows the span waterfall with the full request lifecycle, including nested agent invocations and LLM calls.

Trace span waterfall showing nested agent and LLM call hierarchy

By default, non-GenAI infrastructure spans are hidden. Toggle Show all spans to reveal them. GenAI-related spans (GENERATION, AGENT, TOOL) are expanded automatically while infrastructure spans start collapsed.

Each span has sub-tabs for Overview, Gen AI (model/token/cost summary), Transcript, and Errors.

Scores​

Shows all evaluator scores attached to the trace. Each row displays the score name, value, source (EVAL, ANNOTATION, or API), data type, model used, and comment.

Costs​

Summarizes total cost, tokens, and LLM calls broken down by model. Shows separate breakdowns for model costs and evaluator costs.

Costs tab showing total cost, tokens, and cost by model breakdown

Trace Actions​

The trace detail toolbar includes several actions:

ActionDescription
Analyze with AIOpens the AI sidebar assistant with the trace as context
Open in PlaygroundLoads the trace conversation into the Playground for iteration
Add to DatasetAdds the trace as a test case to a dataset
Open TraceOpens the trace in the standard trace lookup view

Getting Started​

1. Install OpenTelemetry GenAI instrumentation​

The official OpenTelemetry GenAI instrumentation libraries auto-instrument LLM provider SDKs - no manual span creation required. Install the instrumentation package for your LLM provider alongside the core OpenTelemetry SDK and OTLP exporter:

opentelemetry-instrumentation-google-genai>=0.7b1
opentelemetry-exporter-otlp-proto-http>=1.30.0
opentelemetry-sdk>=1.30.0
google-genai>=1.0.0

See the OpenTelemetry GenAI instrumentation registry for a full list of supported providers (Anthropic, AWS Bedrock, LangChain, and more).

2. Instrument your application​

Initialize the OpenTelemetry SDK with OTLP export, then activate the GenAI instrumentor for your provider:

from opentelemetry import _logs as otel_logs
from opentelemetry import trace as otel_trace
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.google_genai import GoogleGenAiSdkInstrumentor
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


def setup_opentelemetry():
    resource = Resource.create({"service.name": "my-llm-app"})

    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    otel_trace.set_tracer_provider(tracer_provider)

    logger_provider = LoggerProvider(resource=resource)
    logger_provider.add_log_record_processor(
        BatchLogRecordProcessor(OTLPLogExporter())
    )
    otel_logs.set_logger_provider(logger_provider)


setup_opentelemetry()
GoogleGenAiSdkInstrumentor().instrument()

After this, all calls to the Google Gemini SDK (e.g. client.models.generate_content(...)) are automatically traced with gen_ai.* attributes.

3. Set environment variables​

The application needs these environment variables:

VariableDescription
OTEL_EXPORTER_OTLP_ENDPOINTOTLP endpoint — points to your OTel Collector (e.g. http://otel-collector:4318)
OTEL_SEMCONV_STABILITY_OPT_INSet to gen_ai_latest_experimental to enable GenAI semantic conventions (not needed for Pydantic AI)
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENTSet to span_and_event to capture prompt/response content (not needed for Pydantic AI — it captures content by default)
caution

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT captures full prompt and response text. This may include sensitive data — review your organization's data policies before enabling in production.

4. Configure the OpenTelemetry Collector​

Route traces and logs from your application to Oodle through an OpenTelemetry Collector. The collector exports to Oodle's OTLP endpoint which supports standard OTLP paths (/v1/traces, /v1/logs):

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  otlphttp/oodle:
    endpoint: "https://<OODLE_INSTANCE>-otlp.collector.oodle.ai"
    headers:
      "X-OODLE-INSTANCE": "<OODLE_INSTANCE>"
      "X-API-KEY": "<OODLE_API_KEY>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/oodle]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/oodle]

Replace <OODLE_INSTANCE> and <OODLE_API_KEY> with your credentials from Settings → API Keys in the Oodle UI (ap1, us1).

See the OpenTelemetry integration guide for full collector and SDK configuration options.

5. Verify in Oodle​

Navigate to GenAI > Traces (ap1, us1). Click on a trace to see the Transcript, Trace waterfall, and Costs breakdown.

Demo Applications​

The oodle-onboarding repository includes ready-to-run demos:

git clone https://github.com/oodle-ai/oodle-onboarding.git
cd oodle-onboarding/llmops-otel-demo
cp .env.example .env
# Edit .env with your OODLE_INSTANCE, OODLE_API_KEY, and GEMINI_API_KEY
make up
make test-chat

Each demo includes an OTel Collector and Docker Compose configuration — everything needed to see LLM traces in Oodle in minutes.

Cost Tracking​

Once model definitions are configured, Oodle shows cost data on every LLM trace. The cost is calculated as:

cost = (input_tokens / 1,000,000) × input_price
     + (output_tokens / 1,000,000) × output_price

For parent spans (agent invocations), Oodle aggregates token counts from child LLM spans and shows the total estimated cost. See the Model Definitions page for pricing configuration.

Support

If you need assistance or have any questions, please reach out to us through: