# Oodle
> Oodle is a fully managed, AI-native observability platform — a drop-in replacement for Prometheus and ELK stack at scale. Logs, metrics, traces, and Kubernetes monitoring at 1/5th the cost with zero management.
This file contains all documentation content in a single document following the llmstxt.org standard.
## Datasets & Experiments
Datasets let you organize test cases for systematic
evaluation of your LLM application. Each dataset
contains input/expected-output pairs that you can run
through an LLM with attached evaluators to measure
quality at scale.
Navigate to **Agent Observability → Datasets**
in the sidebar ().
## Creating a Dataset
1. Click **+ New Dataset**
2. Enter a name (e.g. `sms-dataset-v1`)
3. Optionally upload a CSV file to import items
4. Click **Create**
:::note
Creating and managing datasets requires the
**Editor** or **Admin** role.
:::
### CSV Import
You can bulk-import items from a CSV file (up to
50 MB). The CSV must include an `input` column
(exact case). Optional columns:
| Column | Description |
|--------|-------------|
| `input` | The user query or prompt (required) |
| `expected_output` | The ideal response |
| `metadata` | Additional context as JSON |
A preview of the first few rows is shown before
import. Partial imports are supported — if some
rows fail, the successful rows are still added and
a summary toast shows the error count.
## Adding Items
From a dataset's detail page, click **+ New Item**
to add entries manually or via CSV.
### Manual Entry
Add one or more rows with **Input** (required),
**Expected Output**, and **Metadata** fields. Click
**+ Add another row** to add multiple items at once.
### From a Trace
On any trace detail page, click **Add to Dataset**.
Oodle splits the conversation into input and
expected output and adds it to the dataset you
select. This is useful for capturing real
production examples as test cases.
## Running Experiments
Experiments run your dataset items through an LLM
and optionally score the outputs with evaluators.
1. Open a dataset and click **Run Experiment**
2. Configure the experiment:
| Field | Description |
|-------|-------------|
| **LLM Connection** | Which provider credentials to use ([configure here](/agent-observability/llm-connections)) |
| **Model** | The model to run items against |
| **Evaluators** | One or more active evaluator rules to score outputs |
| **Prompt** | Optional prompt template to wrap inputs (by label or version) |
3. Click **Run Experiment**
The experiment runs asynchronously. Results appear
in the **Runs** tab.
:::tip
If you don't have an LLM connection configured yet,
the form links to **Settings → LLM Connections**
where you can add one.
:::
## Viewing Results
The **Runs** tab shows all experiment runs with
their status, LLM connection, evaluator, prompt
reference, and creation time.
Click any run to open a detail drawer showing:
- **Run metadata** — status, connection, model,
evaluators, prompt, and any errors
- **Per-item results** — a table with Input,
Metadata, Expected Output, Actual Output, and
score badges from each evaluator
While an experiment is running, the drawer polls
for new results every few seconds and shows a
progress indicator.
### Run Statuses
| Status | Meaning |
|--------|---------|
| Pending | Queued, not yet started |
| Running | Processing dataset items |
| Completed | All items processed |
| Failed | An error stopped the run |
---
## Agent Observability
Oodle's Agent Observability helps engineering teams
track how large language models are used across their
applications - including prompts, responses, token
consumption, and estimated costs.
## How It Works
Oodle ingests traces and logs that follow the
[OpenTelemetry Gen AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
When your application instruments LLM calls with these
conventions, Oodle automatically:
- Detects the **model** from `gen_ai.request.model`
and `gen_ai.response.model` span attributes
- Extracts **input and output token counts** from
`gen_ai.usage.input_tokens` and
`gen_ai.usage.output_tokens`
- Displays **prompt and response content** from
`gen_ai.input.messages`, `gen_ai.output.messages`,
and `gen_ai.system_instructions` span attributes
- Calculates **estimated costs** based on your
configured [model pricing](./model-definitions)
All of this is surfaced in the Agent Observability hub
at **Agent Observability → Overview**
().
## Exploring Traces
The Agent Observability hub has several tabs for
different views of your GenAI data.
### Overview
The default tab shows an embedded Grafana dashboard
with key metrics: token usage, cost trends, trace
counts, and model breakdowns over time.
### Traces
The main trace list with columns for timestamp, name,
input, output, model, sentiment, turns, latency, and
cost.
Search traces by **Name**, **Input**, **Output**, or
**Input/Output** combined using the field dropdown
next to the search box.
Click any trace to open the detail view (see
[Trace Detail](#trace-detail) below).
### Agent Graph
An interactive topology graph showing how agents,
tools, and LLM calls connect in your application.
Each node displays request count, error rate, and
average latency. Click a node to see:
- **Decision Paths** — outgoing connections split
into sub-agents and tool calls with probability,
error rate, and latency
- **Traces** — filtered trace list for that agent
- **Insights** — AI-generated recommendations
specific to that agent (if available)
The Agent Graph has its own sidebar with environment
and service filters, plus a node search and
visibility toggle.
### Observations
A span-level table showing individual LLM calls
rather than full traces. Columns include start time,
span type, name, input, output, model, sentiment,
log level, latency, and cost.
### Sessions
Groups traces by `session.id` into multi-turn
conversations. Each row shows the session ID, trace
count, turns, sentiment, total cost, duration, and
last activity time. Click a session to open a detail
drawer with stats and a cross-trace transcript.
:::tip
To enable sessions, set `session.id` as a span
attribute in your instrumentation. Traces sharing
the same session ID are grouped together
automatically.
:::
### Users
Aggregates usage by `user.id` — showing total events,
turns, sentiment, token count, and cost per user.
Click a user to filter the Traces tab to that user's
activity.
:::tip
Set `user.id` as a span attribute to enable
per-user tracking.
:::
### Insights
Appears when Oodle has generated AI-powered
recommendations for your GenAI application. Insights
are categorized by type (Quality, Efficiency,
Resource, Performance) and severity (High, Medium,
Low). Use this tab to identify optimization
opportunities and quality issues.
## Sidebar Filters
The sidebar on the left lets you filter data across
most tabs:
| Filter | Description |
|--------|-------------|
| **Time Range** | Controls the query window |
| **Has Error** | Show only traces with errors |
| **Environment** | Filter by `resource::env` |
| **Service** | Filter by `resource::service.name` |
| **Model** | Filter by `gen_ai.request.model` |
| **Agent Name** | Filter by `gen_ai.agent.name` |
| **Operation Name** | Filter by `gen_ai.operation.name` |
| **User ID** | Filter by `user.id` |
| **Sentiment** | Range slider from −1 to 1 |
| **Score** | Range slider from 0 to 1 |
| **Score Name** | Filter by evaluator score name |
You can also add arbitrary label filters using the
filter bar above the tab content.
## Trace Detail
Clicking on a trace opens the detail view with tabs
for **Transcript**, **Trace**, **Logs**, **Scores**,
and **Costs**.
### Transcript
Displays the full conversation — system instructions,
user prompts, and assistant responses — alongside the
model name and token count.
Use the search bar to find text across all messages.
Matches are highlighted and the view scrolls to
each result.
For traces with multiple agents (identified by
`gen_ai.agent.name`), a **By Agent** toggle groups
the conversation by agent.
### Trace
Shows the span waterfall with the full request
lifecycle, including nested agent invocations and LLM
calls.
By default, non-GenAI infrastructure spans are hidden.
Toggle **Show all spans** to reveal them. GenAI-related
spans (GENERATION, AGENT, TOOL) are expanded
automatically while infrastructure spans start
collapsed.
Each span has sub-tabs for **Overview**, **Gen AI**
(model/token/cost summary), **Transcript**, and
**Errors**.
### Scores
Shows all evaluator scores attached to the trace.
Each row displays the score name, value, source
(EVAL, ANNOTATION, or API), data type, model used,
and comment.
### Costs
Summarizes total cost, tokens, and LLM calls broken
down by model. Shows separate breakdowns for model
costs and evaluator costs.
### Trace Actions
The trace detail toolbar includes several actions:
| Action | Description |
|--------|-------------|
| **Analyze with AI** | Opens the AI sidebar assistant with the trace as context |
| **Open in Playground** | Loads the trace conversation into the [Playground](./playground) for iteration |
| **Add to Dataset** | Adds the trace as a test case to a [dataset](./datasets) |
| **Open Trace** | Opens the trace in the standard trace lookup view |
## Getting Started
### 1. Install OpenTelemetry GenAI instrumentation
The official OpenTelemetry GenAI instrumentation libraries
auto-instrument LLM provider SDKs - no manual span
creation required. Install the instrumentation package
for your LLM provider alongside the core OpenTelemetry
SDK and OTLP exporter:
```txt
opentelemetry-instrumentation-google-genai>=0.7b1
opentelemetry-exporter-otlp-proto-http>=1.30.0
opentelemetry-sdk>=1.30.0
google-genai>=1.0.0
```
```txt
opentelemetry-instrumentation-openai-v2>=2.0b0
opentelemetry-exporter-otlp-proto-http>=1.30.0
opentelemetry-sdk>=1.30.0
openai>=1.0.0
```
```txt
pydantic-ai
opentelemetry-exporter-otlp-proto-http>=1.30.0
opentelemetry-sdk>=1.30.0
```
```txt
traceloop-sdk
google-genai>=1.0.0
```
```txt
openlit
```
See the [OpenTelemetry GenAI instrumentation registry](https://opentelemetry.io/ecosystem/registry/?language=python&component=instrumentation&search=gen+ai)
for a full list of supported providers (Anthropic,
AWS Bedrock, LangChain, and more).
### 2. Instrument your application
Initialize the OpenTelemetry SDK with OTLP export, then
activate the GenAI instrumentor for your provider:
```python
from opentelemetry import _logs as otel_logs
from opentelemetry import trace as otel_trace
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.google_genai import GoogleGenAiSdkInstrumentor
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
def setup_opentelemetry():
resource = Resource.create({"service.name": "my-llm-app"})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
otel_trace.set_tracer_provider(tracer_provider)
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
BatchLogRecordProcessor(OTLPLogExporter())
)
otel_logs.set_logger_provider(logger_provider)
setup_opentelemetry()
GoogleGenAiSdkInstrumentor().instrument()
```
After this, all calls to the Google Gemini SDK (e.g.
`client.models.generate_content(...)`) are automatically
traced with `gen_ai.*` attributes.
```python
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import set_tracer_provider
from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentationSettings
def setup_opentelemetry():
resource = Resource.create({"service.name": "my-llm-app"})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter())
)
set_tracer_provider(tracer_provider)
setup_opentelemetry()
Agent.instrument_all(InstrumentationSettings(
use_aggregated_usage_attribute_names=False,
))
```
Pydantic AI captures prompt and response content by
default. Pass `include_content=False` to
`InstrumentationSettings` to disable.
[Traceloop's OpenLLMetry](https://github.com/traceloop/openllmetry)
SDK auto-instruments supported LLM SDKs with a single
init call. No manual span or provider setup is required:
```python
from google import genai
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow
# Auto-instruments Gemini, OpenAI, Anthropic, LangChain, and more.
Traceloop.init(app_name="my-llm-app")
client = genai.Client()
# Optional: group related LLM calls under a named workflow span.
@workflow(name="chat")
def chat(message: str) -> str:
response = client.models.generate_content(
model="gemini-flash-latest",
contents=message,
)
return response.text
```
Traceloop captures prompt and response content by
default. Set `TRACELOOP_TRACE_CONTENT=false` to disable.
:::note
Traceloop uses its own environment variable instead of
the `OTEL_*` ones below. Point the SDK at your collector
with `TRACELOOP_BASE_URL=http://otel-collector:4318`.
Content capture is on by default; disable it with
`TRACELOOP_TRACE_CONTENT=false`.
:::
[OpenLIT](https://github.com/openlit/openlit) auto-instruments
supported LLM providers, vector databases, MCP tools, and GPUs
with a single `init()` call. No manual span creation needed:
```python
openlit.init(
service_name="my-llm-app",
otlp_endpoint="http://otel-collector:4318",
capture_message_content=True,
)
```
Set `capture_message_content=True` to capture prompt and
response content in span attributes. Review your
organization's data policies before enabling in production.
:::note
OpenLIT does not require any `OTEL_*` environment
variables. The `otlp_endpoint` and
`capture_message_content` parameters are passed directly
to `openlit.init()`.
:::
### 3. Set environment variables
The application needs these environment variables:
| Variable | Description |
|----------|-------------|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP endpoint — points to your OTel Collector (e.g. `http://otel-collector:4318`) |
| `OTEL_SEMCONV_STABILITY_OPT_IN` | Set to `gen_ai_latest_experimental` to enable GenAI semantic conventions (not needed for Pydantic AI) |
| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | Set to `span_and_event` to capture prompt/response content (not needed for Pydantic AI — it captures content by default) |
:::caution
`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` captures full prompt and response text. This may include sensitive data — review your organization's data policies before enabling in production.
:::
### 4. Configure the OpenTelemetry Collector
Route traces and logs from your application to Oodle
through an OpenTelemetry Collector. The collector
exports to Oodle's OTLP endpoint which supports standard
OTLP paths (`/v1/traces`, `/v1/logs`):
```yaml
receivers:
otlp:
protocols:
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
timeout: 5s
send_batch_size: 512
exporters:
otlphttp/oodle:
endpoint: "https://-otlp.collector.oodle.ai"
headers:
"X-OODLE-INSTANCE": ""
"X-API-KEY": ""
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/oodle]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/oodle]
```
Replace `` and `` with your
credentials from **Settings → API Keys** in the Oodle UI
().
See the [OpenTelemetry integration guide](/integrations/traces/otel)
for full collector and SDK configuration options.
### 5. Verify in Oodle
Navigate to **GenAI > Traces** (). Click on
a trace to see the Transcript, Trace waterfall, and Costs
breakdown.
## Demo Applications
The [oodle-onboarding](https://github.com/oodle-ai/oodle-onboarding)
repository includes ready-to-run demos:
- [**llmops-otel-demo**](https://github.com/oodle-ai/oodle-onboarding/tree/main/llmops-otel-demo) — Google Gemini with OTel instrumentation
- [**pydantic-ai-demo**](https://github.com/oodle-ai/oodle-onboarding/tree/main/pydantic-ai-demo) — Pydantic AI multi-agent example with tool use and structured outputs
- [**traceloop-demo**](https://github.com/oodle-ai/oodle-onboarding/tree/main/traceloop-demo) — Traceloop (OpenLLMetry) SDK auto-instrumenting Google Gemini
- [**openlit-demo**](https://github.com/oodle-ai/oodle-onboarding/tree/main/openlit-demo) — OpenLIT with agents (LangGraph), guardrails, VectorDB (ChromaDB RAG), and MCP tool tracing
```bash
git clone https://github.com/oodle-ai/oodle-onboarding.git
cd oodle-onboarding/llmops-otel-demo
cp .env.example .env
# Edit .env with your OODLE_INSTANCE, OODLE_API_KEY, and GEMINI_API_KEY
make up
make test-chat
```
Each demo includes an OTel Collector and Docker Compose
configuration — everything needed to see LLM traces in
Oodle in minutes.
## Cost Tracking
Once [model definitions](./model-definitions) are
configured, Oodle shows cost data on every LLM trace.
The cost is calculated as:
```
cost = (input_tokens / 1,000,000) × input_price
+ (output_tokens / 1,000,000) × output_price
```
For parent spans (agent invocations), Oodle aggregates
token counts from child LLM spans and shows the total
estimated cost. See the [Model Definitions](./model-definitions)
page for pricing configuration.
---
## LLM Connections
LLM Connections store your provider API credentials
so Oodle can call LLMs on your behalf. They are
required for the
[Playground](/agent-observability/playground),
[LLM-as-Judge evaluators](/evaluators/llm-as-judge),
and [dataset experiments](/agent-observability/datasets).
Navigate to **Settings → LLM Connections**
().
:::note
Only **Admin** users can create and manage LLM
connections.
:::
## Supported Providers
| Provider | Default Base URL |
|----------|------------------|
| **Google** | `https://generativelanguage.googleapis.com` |
| **Anthropic** | `https://api.anthropic.com` |
| **OpenAI** | `https://api.openai.com/v1` |
## Creating a Connection
1. Click **+ New Connection**
2. Fill in the required fields:
| Field | Required | Description |
|-------|----------|-------------|
| **Name** | Yes | Display name for the connection |
| **Provider** | Yes | Google, Anthropic, or OpenAI |
| **API Key** | Yes | Your provider API key |
| **Default Model** | No | Model used when none is explicitly selected |
| **Base URL** | No | Override the default API endpoint (useful for proxies or custom deployments) |
| **Custom Models** | No | Additional model names not in the built-in list (e.g. `deepseek/deepseek-v4-flash`) |
| **Custom Headers** | No | Extra HTTP headers sent with every request |
3. Optionally toggle **Set as default connection**
to make this the primary connection
4. Optionally toggle **Show default models from
provider** off to hide built-in models and only
show your custom models
5. Click **Create**
## Default Connection
One connection can be marked as the default. The
Playground, evaluator wizard, and experiment form
auto-select the default connection when no other
is specified.
## Custom Models
If your provider supports models not in Oodle's
built-in list, add them as **Custom Models**. These
appear alongside built-in models in every model
selector. You can also type any model name directly
in model dropdowns — select **Use "model-name"** to
use an unlisted model.
## Editing and Deleting
Click the edit icon on any connection to update its
configuration. When editing, leave the API key
blank to keep the existing key.
:::warning
Deleting a connection stops any evaluators and
Playground sessions that use it.
:::
---
## Model Definitions
Model Definitions let you configure pricing
information for each LLM model your applications use.
Navigate to **Agent Observability → Model Definitions**
in the sidebar ().
## How pricing works
Each model definition includes:
| Field | Description |
|-------|-------------|
| **Model Name** | Display name (e.g. `gemini-flash-latest`) |
| **Match Pattern** | Regex matched against the model name in traces |
| **Input Price** | Cost per 1M input tokens (USD) |
| **Output Price** | Cost per 1M output tokens (USD) |
When a trace span includes token usage and a model
name, Oodle walks the model definitions list and
applies the first matching pattern to estimate the
cost.
## Adding a model
1. Go to **Agent Observability → Model Definitions**
()
2. Models detected in your traces that don't match any
existing pattern appear at the top of the table with
a warning - click to add pricing
3. Enter the model name, a regex match pattern, and
input/output prices per 1M tokens
---
## Playground
The Playground is an interactive testing environment
for LLM prompts. Compare outputs across models
side by side, define tools for function calling,
and iterate on prompts before deploying them.
Navigate to **Agent Observability → Playground**
in the sidebar ().
:::info Beta
The Playground is currently in Beta.
:::
## Prerequisites
The Playground requires at least one
[LLM Connection](/agent-observability/llm-connections)
to be configured. If no connections exist, the model
selector shows a link to **Settings → LLM
Connections**.
## Getting Started
1. Select an **LLM Connection** and **Model** from
the dropdowns at the top of the window
2. Add messages using the **+ Message** dropdown
(User, Assistant, System, Developer, or Tool)
3. Type your prompt content in the message editor
4. Click **Submit** or press **⌘ Enter** to run
The output appears at the bottom of the window
with the model's response, including any tool calls
or reasoning output.
## Multi-Window Comparison
Click **New split window** to add up to four
side-by-side windows. Each window has its own
connection, model, messages, and output — useful for
A/B testing prompts across different models or
configurations.
- **Duplicate window** — copies the window
configuration
- **Remove window** — removes a window (available
when more than one is open)
- **Run All** (⌘ Enter) — submits all windows that
have a model selected, in parallel
- **Reset** — replaces all windows with a single
empty window
## Messages
Messages support the following roles:
| Role | Purpose |
|------|---------|
| **System** | System instructions for the model |
| **User** | User input / prompt |
| **Assistant** | Model response (for few-shot examples) |
| **Developer** | Developer instructions |
| **Tool** | Tool result linked to a tool call |
Click the role badge on any message to cycle through
roles. Drag the handle to reorder messages.
### Prompt References
Type `@@@` in a message to link to a saved
[prompt](/prompt-management). Oodle resolves prompt
references before sending the request to the model.
## Template Variables
Use `{{variable_name}}` syntax in message content
to create template variables. The **Variables**
panel (badge shows count) lets you fill in values
for each variable before running.
All variables must have values before submitting —
the Playground warns if any are unfilled.
## Tools (Function Calling)
Click **Tools** to define function tools the model
can call. Each tool has:
- **Name** — e.g. `get_weather`
- **Description** — what the tool does
- **JSON Schema** — parameter schema (visual
builder or raw JSON)
When the model responds with tool calls, they appear
as expandable blocks in the output.
:::caution
Tools and structured output cannot be used at the
same time.
:::
## Structured Output
Click **Schema** to define a JSON schema that
constrains the model's output format. Specify a
name, description, and schema using the visual
builder or raw JSON editor.
## Model Parameters
Click the gear icon to configure optional
parameters:
| Parameter | Range | Default |
|-----------|-------|---------|
| **Temperature** | 0–2 | 1.0 |
| **Max Tokens** | 1–65536 | 2048 |
| **Top P** | 0–1 | 1.0 |
| **Additional options** | Provider-specific JSON | `{}` |
Each parameter has a toggle — enable it to override
the model's default.
## Seeding from Traces
From any trace detail page, click **Open in
Playground** to load the trace's conversation into
the Playground. This seeds:
- All messages (system, user, assistant, tool)
- The model name
- Tool definitions (if present in the trace)
Use this to replay and iterate on real production
conversations.
## Seeding from Prompts
From any prompt detail page, click **Test in
Playground** to load the prompt content into the
Playground. Chat prompts load as multi-message
conversations; text prompts load as a single system
message.
## Save as Prompt
Click **Save as Prompt** in the toolbar to save the
current messages as a new prompt. This opens the
prompt creation page with the messages pre-filled.
## Streaming
The **Stream** checkbox (enabled by default)
controls whether responses stream token by token or
arrive as a complete response. Streaming shows a
**Stop** button to end generation early.
---
## Agents
Oodle Agents allow you to securely connect your private infrastructure to
Oodle. An agent runs inside your network (typically as a Kubernetes
deployment) and establishes an outbound WebSocket connection to Oodle.
This tunnel lets Oodle reach resources that are not publicly
accessible, such as internal APIs, databases, and services behind a firewall.
The agent is open source: [github.com/oodle-ai/oodle-agent](https://github.com/oodle-ai/oodle-agent).
## Use Cases
| Use Case | Description |
|----------|-------------|
| **Private endpoint monitoring** | Run [synthetic checks](/synthetic-monitoring) against internal HTTP endpoints, DNS, TCP, Ping, SSL, and Traceroute targets that are not reachable from the public internet |
| **Private database datasources** | Connect Grafana to Postgres databases inside your VPC without exposing them publicly |
## How It Works
1. You create an agent in the Oodle UI and receive a one-time registration
token
2. You deploy the agent into your Kubernetes cluster using Helm
3. On first connection the agent uses the registration token to obtain a
client certificate via mTLS
4. The agent opens a persistent WebSocket connection to Oodle
5. Oodle routes requests (synthetic checks, database queries) through the
tunnel to your private infrastructure
All traffic is encrypted in transit using mutual TLS (mTLS). The agent
authenticates with a client certificate, and Oodle authenticates via its
server certificate pinned to the Amazon CA.
## Creating an Agent
:::note
Creating an agent requires the **Admin** role. Users with the Editor or
Viewer role cannot create or manage agents.
:::
1. Navigate to **Agents** in the sidebar
2. Click **Create Agent**
3. Enter a descriptive name (e.g. `production-cluster`, `staging-vpc`)
4. Click **Create**
After creation, you will see two commands that must be run in order. **Save
the registration token immediately** — it is single-use and expires
in 24 hours.
### Step 1: Create the Kubernetes Secret
The first command creates a namespace and a Kubernetes secret containing the
agent's one-time registration token:
```bash
kubectl get namespace oodle-monitoring 2>/dev/null || \
kubectl create namespace oodle-monitoring
kubectl create secret generic oodle-agent- \
--namespace oodle-monitoring \
--from-literal=registration-token=
```
### Step 2: Install the Agent via Helm
The second command deploys the agent using the official
[Helm chart](https://github.com/oodle-ai/oodle-agent):
```bash
helm repo add oodle-ai https://oodle-ai.github.io/helm-charts
helm repo update
helm upgrade --install oodle-agent oodle-ai/oodle-agent \
--namespace oodle-monitoring \
--create-namespace \
--wait \
--set oodle.instance= \
--set oodle.agentId= \
--set oodle.agentName= \
--set existingSecret=oodle-agent- \
--set oodle.gatewayUrls=wss:///ws
```
:::tip
Both commands are pre-filled with the correct values in the UI. You can copy
them directly.
:::
:::info
On its first connection, the agent uses the registration token to submit a
Certificate Signing Request (CSR) and receives a signed client certificate.
All subsequent connections use mutual TLS (mTLS) — the registration
token is automatically consumed and no longer needed.
:::
### Step 3: Restrict Allowed Endpoints (Optional)
By default the agent can reach any endpoint in your network. To restrict which
hosts the agent is allowed to connect to, add the `policy` values to the Helm
install command:
```bash
helm repo add oodle-ai https://oodle-ai.github.io/helm-charts
helm repo update
helm upgrade --install oodle-agent oodle-ai/oodle-agent \
--namespace oodle-monitoring \
# ... same flags as above ...
--set policy.enable_access_restrictions=true \
--set "policy.allowed_endpoints[0]=my-postgres.internal:5432" \
--set "policy.allowed_endpoints[1]=*.prod.internal:443" \
--set "policy.allowed_endpoints[2]=10.0.1.50:8080"
```
Supported endpoint patterns:
| Pattern | Description |
|---------|-------------|
| `host:port` | Exact match (e.g. `my-db.internal:5432`) |
| `host` | Any port on that host |
| `*.example.com:443` | Wildcard subdomain with specific port |
| `*:5432` | Any host on a specific port |
The policy is enforced for **all** protocols: HTTP, Postgres, TCP, Ping,
DNS, SSL, and Traceroute.
You can also restrict which Kubernetes namespaces and resources the agent can
read:
```bash
--set "policy.kubernetes.allowed_namespaces[0]=production" \
--set "policy.kubernetes.allowed_namespaces[1]=monitoring" \
--set "policy.kubernetes.deny_resources[0]=secrets"
```
To update the policy after installation, run `helm upgrade` with the new
values, or edit the `oodle-agent-policy` ConfigMap directly and restart the
agent pod.
## Viewing Agent Status
The Agents list page displays all registered agents with:
- **Name**: The agent name and its unique ID
- **Status**: Current connection status (`ACTIVE` or `DISCONNECTED`)
- **Created**: When the agent was registered
- **Last Heartbeat**: The most recent heartbeat received from the agent
Click on any agent row to open the detail drawer.
## Agent Detail Drawer
### Overview Tab
Shows creation time, last heartbeat, capabilities, and an embedded health
dashboard with real-time metrics for the agent.
### Datasources Tab
Manage Postgres datasources that are routed through this agent. Each
datasource creates a corresponding Grafana Postgres datasource that
tunnels queries through the agent to a private database.
#### Adding a Postgres Datasource
1. Open the agent detail drawer and go to the **Datasources** tab
2. Click **Add Datasource**
3. Fill in the connection details:
- **Datasource Name**: A human-readable name
- **Host**: The private Postgres host and port
(e.g. `my-db.internal:5432`)
- **Database**: The database name
- **User / Password**: Postgres credentials
- **SSL Mode**: `disable`, `require`, `verify-ca`, or `verify-full`
- **TLS Certificates**: (Optional) CA cert, client cert, and client
key for mTLS
4. Click **Create**
Once created, a Grafana Postgres datasource is automatically provisioned.
You can click **Explore** to query the database directly from Grafana, or
use **Save & Test** to verify connectivity.
### Uninstall Tab
Provides step-by-step commands to cleanly remove the agent:
1. **Uninstall the Helm release**:
```bash
helm uninstall oodle-agent \
--namespace oodle-monitoring
```
2. **Delete the Kubernetes secret**:
```bash
kubectl delete secret oodle-agent- \
--namespace oodle-monitoring
```
3. **Delete the agent from Oodle** using the **Delete Agent** button
:::caution
Deleting an agent disconnects all datasources and synthetic monitors
routed through it.
:::
## Best Practices
### Naming Conventions
Use names that identify the environment and cluster, e.g.:
- `prod-us-east-1`
- `staging-eks`
- `dev-local`
### High Availability
The agent maintains multiple connections for redundancy. If one connection
becomes unavailable, traffic is automatically routed through another.
### Network Requirements
The agent only requires **outbound** access on port 443 (WebSocket over
TLS). No inbound firewall rules are needed.
---
## AI Canvas
AI Canvas is a full-page, conversational AI assistant built into Oodle.
It can query your metrics, logs, and traces, create Grafana dashboards
and alerts, navigate the UI on your behalf, and answer questions about
your infrastructure — all through natural language.
## Accessing AI Canvas
There are two ways to interact with the AI assistant:
### Full-Page Canvas
Click **AI Assistant** in the sidebar, then select **AI Canvas** to open
the dedicated full-page experience. You can also navigate directly to
`/ai-canvas`. The full-page canvas includes a chat history sidebar on the
left so you can revisit previous conversations.
### Sidebar Assistant
Click **AI Assistant** (or press Cmd + I) from any
page to open the sidebar panel. The sidebar overlays the current page,
giving you quick access to the AI while keeping your current context
visible.
## Capabilities
| Capability | Description |
|------------|-------------|
| **Metrics** | Find metrics, query with PromQL, and visualize results inline |
| **Logs** | Search and summarize logs, identify error patterns |
| **Traces** | Query traces and analyze latency or error patterns |
| **Dashboards** | Create and update Grafana dashboards from a prompt |
| **Alerts** | Create, update, and explain alert monitors |
| **Anomalies** | Query detected anomalies across your infrastructure |
| **Navigation** | Navigate to pages, adjust time ranges, and fill forms |
| **Integrations** | Answer setup questions using Oodle documentation |
## Home Page
When you open AI Canvas without an active conversation,
the home page shows contextual information below the chat
input to help you get started:
- **Recent Dashboards** — your most-visited Grafana
dashboards from the past week, sorted by frequency.
If you have fewer than ten personal dashboards, the
list is filled with popular dashboards across your
organization. Click any dashboard to open it directly.
- **Anomalies** — the top anomalies detected across
your infrastructure in the last 24 hours, sorted by
severity and recency. Click any anomaly to jump to
the anomalies page with that item expanded.
Both section headers link to their respective full pages
(Dashboards and Anomalies).
## Using AI Canvas
### Starting a Conversation
Type your question in the input box at the bottom and press
Enter. You can also click one of the suggested
prompts to get started.
Example prompts:
- "Create dashboard for high CPU usage by container"
- "Search error logs for last 15 min"
- "Create an alert for CPU throttling"
- "Summarize logs for last 15 mins"
- "How do I send k8s logs to Oodle?"
### Tool Calls and Inline Results
The AI assistant uses tools to interact with your data. As it works, you
will see expandable tool call cards showing what the assistant is doing —
for example, finding metrics, running PromQL queries, or saving a
dashboard. Query results such as charts and tables are rendered inline in
the conversation.
### Auto-Approve Actions
Some actions (like creating an alert or navigating to a page) require
approval before executing. Enable the **Auto-approve actions** checkbox
at the bottom of the chat to let the assistant execute these actions
without prompting.
### Following Up
After the assistant responds, you can ask follow-up questions in the
same conversation. For example, after creating a dashboard you might ask
the assistant to add more panels or adjust the time range.
## Chat History
AI Canvas persists your conversations. The **Chat History** sidebar on
the left lists all previous sessions with their title, age, and message
count. Click any entry to reload that conversation and continue where you
left off. Click **Start New Chat** to begin a fresh session.
## Example Workflows
### Create a Dashboard
> "Create a health dashboard for the frontend service"
The assistant will:
1. Search for relevant metrics in your environment
2. Run PromQL queries to verify the metrics return data
3. Build a Grafana dashboard with appropriate panels
4. Save the dashboard and provide a link to open it
### Debug an Alert
> "Why is my CPU throttling alert firing?"
The assistant will:
1. Look up the alert definition and current status
2. Query the underlying metrics to identify the source
3. Check logs and traces for correlated errors
4. Summarize findings and suggest next steps
### Search Logs
> "Show me error logs from the payment service in the last hour"
The assistant will:
1. Identify the appropriate log index
2. Run a filtered query for error-level logs
3. Summarize the top patterns and provide representative log entries
---
## Claude
Claude ships with built-in OpenTelemetry support across
all of its coding surfaces: the Claude Code CLI, the
desktop app, and Cowork. Once enabled, Oodle collects
metrics and event logs and turns them into use-case
dashboards covering adoption,
cost and ROI, engineering impact, tool behavior, and
efficiency across your organization.
## Getting Started
### 1. Enable Telemetry
The fastest way is to use the integration tile in the
Oodle UI:
1. Navigate to **Settings → Integrations**
2. Open the **AI Observability** section
3. Click the **Claude Observability** tile
4. Select an API key and follow the steps shown
The tile generates a ready-to-use configuration and lets
you choose how to distribute it: to a single developer,
to everyone working in a repository, or to your whole
organization. Replace ``, ``, and
`` in the snippets below with values from the
integration tile.
#### Individual
For a single developer, export the variables directly in
your shell before running `claude`:
```bash
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=https://
export OTEL_EXPORTER_OTLP_HEADERS="X-API-KEY=, X-OODLE-INSTANCE="
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
export OTEL_LOG_USER_PROMPTS=1
export OTEL_LOG_TOOL_DETAILS=1
# Optional: resource attributes for team/repo breakdowns
export OTEL_RESOURCE_ATTRIBUTES="git.repository.name=my-repo,org.team.name=platform,org.department=engineering,org.cost_center=eng-123"
```
:::tip
For faster feedback during initial setup, lower the
export intervals:
```bash
export OTEL_METRIC_EXPORT_INTERVAL=10000
export OTEL_LOGS_EXPORT_INTERVAL=5000
```
:::
#### Repository
To share telemetry settings with everyone working in a
repository, commit them to `.claude/settings.json` in the
project. Claude Code applies them for anyone who works in
the repo, with no per-developer setup required:
```json
{
"env": {
"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
"OTEL_METRICS_EXPORTER": "otlp",
"OTEL_LOGS_EXPORTER": "otlp",
"OTEL_TRACES_EXPORTER": "otlp",
"CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
"OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
"OTEL_EXPORTER_OTLP_ENDPOINT": "https://",
"OTEL_EXPORTER_OTLP_HEADERS": "X-API-KEY=, X-OODLE-INSTANCE=",
"OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE": "delta",
"OTEL_LOG_USER_PROMPTS": "1",
"OTEL_LOG_TOOL_DETAILS": "1",
"OTEL_RESOURCE_ATTRIBUTES": "git.repository.name=my-repo,org.team.name=platform,org.department=engineering,org.cost_center=eng-123"
}
}
```
#### Server-managed (organization-wide)
To distribute settings across your whole organization
without device management (MDM), use the
[Claude Code admin console](https://claude.ai/admin-settings/claude-code).
Paste the same `env` configuration into **Admin Settings →
Claude Code → Managed settings**. Clients fetch it when
users authenticate and poll for updates hourly.
- Requires a Claude for Teams or Enterprise plan, and the
Owner or Primary Owner role to edit.
- OpenTelemetry settings take effect after a full Claude
Code restart.
See the
[server-managed settings documentation](https://code.claude.com/docs/en/server-managed-settings)
for delivery, caching, and precedence details.
### 2. Verify Data
Once telemetry starts flowing (metrics export every
60 s, events every 5 s by default), navigate to
**AI Cost Management → Claude** in the Oodle
sidebar.
## Dashboards
The Claude analysis page is organized around the
questions engineering leadership asks of a coding agent.
Each tab embeds a dashboard that leads with headline
numbers, followed by trends and ranked breakdowns.
### Usage and Adoption
Is the team using it?
| Panel | Description |
|-------|-------------|
| **Active Users** | Distinct engineers who ran Claude |
| **Sessions** | Sessions started, plus sessions per active user |
| **Active Time** | CLI processing time vs user interaction time |
| **Surface / Model / Terminal Mix** | Where and how sessions run (CLI, desktop, Cowork; model share; editor terminals) |
| **Version Adoption** | Activity by CLI version, to track upgrade velocity |
| **Top Users** | Engineers ranked by sessions and active time |
### Cost and ROI
What are we paying, and what do we get for it?
| Panel | Description |
|-------|-------------|
| **Total Spend** | Estimated USD spend, the headline budget number |
| **Cost per Active User** | Average spend per engineer, for seat-cost modeling |
| **Cost per Commit / Cost per PR** | Unit economics of shipped output |
| **Sessions Producing Outcomes** | Share of sessions that led to a commit or PR |
| **Top Spenders** | Users and sessions ranked by spend |
| **Spend Breakdown** | Spend by model, query source, agent, and skill |
| **Team and Repository Breakdown** | Spend by `org.team.name` and `git.repository.name` resource attributes |
### Engineering Impact
What did it produce?
| Panel | Description |
|-------|-------------|
| **Lines Added / Removed** | Code throughput from accepted suggestions |
| **Commits and Pull Requests** | Shipped output over time |
| **Edit Accept Rate** | Share of suggested edits engineers accepted, with health thresholds |
| **Avg Lines per Session** | Output volume per session |
| **Output Leaderboards** | Top users by lines added, commits, and PRs |
### Tools and Automation
Is the agent working well?
| Panel | Description |
|-------|-------------|
| **Tool Calls** | Invocation volume, distinct tools, and per-tool trends |
| **Tool Duration** | Average execution time by tool |
| **Result Size** | Bytes returned per tool, the main driver of context growth and cost |
| **Bash CLI Breakdown** | Specific commands run through Bash, by count and output size |
| **MCP Servers and Tools** | External integration usage and latency |
| **Skills** | Invocations of codified team workflows |
### Efficiency
Is spend being used well?
| Panel | Description |
|-------|-------------|
| **Cache Hit Rate** | Share of context served from cache, with health thresholds |
| **Token Split** | Input vs cache read vs cache creation volume |
| **Ghost Cost Leaderboard** | Sessions ranked by cache re-priming relative to new input |
| **Cost per Line of Code** | Output per dollar, overall and by model |
| **Prompt Length Analysis** | Prompt size distribution and cost per prompt |
## Sessions
The Sessions tab shows individual Claude sessions:
| Column | Description |
|--------|-------------|
| **Start Time** | When the session began |
| **User** | Email of the developer |
| **Model** | Primary model used |
| **Prompts** | Number of user prompts |
| **Duration** | Wall-clock duration |
| **Cost** | Total estimated cost |
| **Tools** | Number of tool calls |
| **Tokens** | Total tokens (input + output) |
| **Errors** | Count of errors |
Click any row to open a **Session Detail** drawer
showing a turn-by-turn timeline of every event.
### Session Detail Drawer
The drawer displays:
- **Session metadata**: user, model, app version,
terminal type
- **Aggregated stats**: total cost, tokens, tool
calls, errors, duration
- **Turn-by-turn timeline**: each turn is collapsible
and shows the user prompt followed by individual
events (API requests, tool calls, decisions). Every
event row is expandable to reveal the full raw JSON
payload.
## What Gets Collected
### Metrics
Claude exports the following as OpenTelemetry
metrics (delta temporality):
| Metric | Key Labels | Description |
|--------|------------|-------------|
| `claude_code_token_usage` | `type`, `model`, `user_email` | Token count by type (input, output, cacheRead, cacheCreation) |
| `claude_code_cost_usage` | `model`, `user_email` | Estimated cost in USD |
| `claude_code_session_count` | `user_email` | Sessions started |
| `claude_code_active_time_total` | `type` | User and CLI active time in seconds |
| `claude_code_lines_of_code_count` | `type` | Lines added / removed |
| `claude_code_commit_count` | `user_email` | Commits created |
| `claude_code_pull_request_count` | `user_email` | Pull requests created |
| `claude_code_code_edit_tool_decision` | `decision` | Edit accept / reject decisions |
All metrics also carry `session_id`, `service_version`,
`terminal_type`, and `job` (the surface: CLI, desktop app,
or Cowork). Tool call counts, durations, and result sizes
are derived from the `claude_code.tool_result` events
below and power the Tools and Automation dashboard.
### Events (Logs)
Events are exported via the OpenTelemetry logs
protocol. Each event has a `log.attributes.event_type`
field:
| Event Type | Key Attributes |
|------------|----------------|
| `claude_code.user_prompt` | Prompt text (opt-in), character count |
| `claude_code.api_request` | Model, cost, input/output tokens, cache tokens, duration |
| `claude_code.tool_result` | Tool name, success, duration |
| `claude_code.tool_use_decision` | Decision (accept/reject), source |
Events are grouped by `session_id` to reconstruct the
turn-by-turn timeline on the Sessions tab.
## Multi-team Organization Support
Organizations with multiple teams or departments can add custom
[resource attributes](https://code.claude.com/docs/en/monitoring-usage#multi-team-organization-support)
to segment telemetry by repository, team, department, or cost center.
### Recommended Attributes
| Attribute | Purpose | Example |
|-----------|---------|---------|
| `git.repository.name` | Per-repository breakdowns | `my-service` |
| `org.team.name` | Team-level filtering | `platform` |
| `org.department` | Department or business unit | `engineering` |
| `org.cost_center` | Charge-back / cost allocation | `eng-123` |
### Configuration
Set `OTEL_RESOURCE_ATTRIBUTES` as a comma-separated list of
key=value pairs:
```bash
export OTEL_RESOURCE_ATTRIBUTES="git.repository.name=my-repo,org.team.name=platform,org.department=engineering,org.cost_center=eng-123"
```
Or add it to your managed settings file (`.claude/settings.json`):
```json
{
"env": {
"OTEL_RESOURCE_ATTRIBUTES": "git.repository.name=my-repo,org.team.name=platform,org.department=engineering,org.cost_center=eng-123"
}
}
```
:::warning Formatting requirements
Values must **not** contain spaces. Use underscores, camelCase,
or percent-encoding for special characters. See the
[Claude Code docs](https://code.claude.com/docs/en/monitoring-usage#multi-team-organization-support)
for the full specification.
:::
Once set, these attributes appear on all metrics and events,
allowing you to:
- Filter dashboards by team or department
- Track costs per cost center
- Create per-repository usage reports
- Set up team-specific alerts
## Further Reading
- [Claude Code Monitoring Documentation](https://code.claude.com/docs/en/monitoring-usage)
- [OpenTelemetry Integration](/integrations/logs/otel)
- [Claude Code MCP Integration](/integrations/mcp/claude)
---
## OpenAI Codex
OpenAI Codex has native OpenTelemetry support. Once
enabled, Oodle collects metrics and event logs and
turns them into use-case dashboards covering adoption,
token consumption, tool behavior, and performance and
reliability across your organization.
## Getting Started
### 1. Enable Telemetry
The fastest way is to use the integration tile in the
Oodle UI:
1. Navigate to **Settings → Integrations**
2. Open the **AI Observability** section
3. Click the **Codex Observability** tile
4. Select an API key and follow the steps shown
Alternatively, add the following to your Codex
configuration file (`~/.codex/config.toml`):
```toml
#:schema https://developers.openai.com/codex/config-schema.json
[otel]
environment = "production"
log_user_prompt = true
exporter = { otlp-http = {
endpoint = "https:///ingest/otel/v1/logs",
protocol = "binary",
headers = {
"X-API-KEY" = "",
"X-OODLE-INSTANCE" = "",
},
}}
metrics_exporter = { otlp-http = {
endpoint = "https:///v2/otlp/metrics/",
protocol = "binary",
headers = {
"X-API-KEY" = "",
"X-OODLE-INSTANCE" = "",
},
}}
```
Replace ``, ``, and
`` with values from the integration tile.
:::tip
The integration tile in the Oodle UI generates a
ready-to-copy `config.toml` with the correct
endpoints and API key pre-filled.
:::
#### Alternative: Environment Variables
For quick testing you can export the standard
OpenTelemetry variables directly in your shell:
```bash
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https:///v2/otlp/metrics/
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https:///ingest/otel/v1/logs
export OTEL_EXPORTER_OTLP_HEADERS="X-API-KEY=, X-OODLE-INSTANCE="
export OTEL_LOG_USER_PROMPTS=1
export OTEL_METRIC_EXPORT_INTERVAL=10000
export OTEL_LOGS_EXPORT_INTERVAL=5000
```
### 2. Roll Out to Your Team
Commit the `config.toml` above to your dotfiles
repository or place it in each developer's
`~/.codex/` directory via your configuration
management tool.
### 3. Verify Data
Once telemetry starts flowing, navigate to
**AI Cost Management → Codex** in the Oodle sidebar.
## Dashboards
The Codex analysis page is organized around the
questions engineering leadership asks of a coding
agent. Each tab embeds a dashboard that leads with
headline numbers, followed by trends and ranked
breakdowns.
### Usage and Adoption
Is the team using it?
| Panel | Description |
|-------|-------------|
| **Active Users** | Distinct engineers who ran Codex |
| **Conversations** | Distinct Codex conversations, plus conversations per active user |
| **Prompts** | User prompts submitted |
| **Activity Trends** | Active users and conversations over time |
| **Surface / Model / Version Mix** | Where activity runs: the IDE/app backend vs the terminal CLI, model share, and version adoption |
| **Top Users** | Engineers ranked by prompts and tokens |
### Token Consumption
Token volume is the primary driver of Codex spend, so
this tab is the budget view.
| Panel | Description |
|-------|-------------|
| **Total Tokens** | Aggregate token usage, the headline consumption number |
| **Tokens per Active User / per Conversation** | Unit consumption for capacity and seat modeling |
| **Cached Input Rate** | Share of input context served from cache, with health thresholds. Higher is cheaper |
| **Token Breakdown** | Usage by token type (input, output, cached, reasoning), model, surface, and version |
| **Top Users and Sessions** | Highest-consumption engineers and conversations |
| **Prompt Economics** | Average prompt size and total tokens consumed per prompt |
Use the **Token Type** dashboard filter to isolate a
category (input, output, cached_input,
reasoning_output).
### Tools and Automation
Is the agent working well?
| Panel | Description |
|-------|-------------|
| **Tool Calls** | Invocation volume and distinct tools used |
| **Tool Success Rate** | Share of tool calls that succeeded, with health thresholds |
| **Avg Tool Duration** | Mean execution time, overall and per tool |
| **Per-tool Trends and Leaderboards** | Calls, failures, and latency ranked by tool |
| **MCP Servers** | External integration usage grouped by MCP server |
### Performance and Reliability
Codex exports rich latency telemetry, so this tab
answers how fast and stable the agent feels.
| Panel | Description |
|-------|-------------|
| **Avg Time to First Token** | How long users wait before Codex starts responding, with health thresholds |
| **Avg Turn Duration** | End-to-end latency per turn |
| **WebSocket Success Rate** | API transport health, with health thresholds |
| **Latency Trends** | Time to first token / first message and turn duration over time |
| **Startup Latency** | Prewarm and shell snapshot times from launch to a usable session |
| **Per-model Latency** | Time to first token and turn duration ranked by model |
This tab is built on Codex's native metrics, which
carry model, originator, and version dimensions; use
its dashboard filters to slice by those.
## Sessions
The Sessions tab shows individual Codex sessions:
| Column | Description |
|--------|-------------|
| **Start Time** | When the session began |
| **Session** | Codex conversation ID |
| **User** | Email of the developer |
| **Surface** | Where Codex ran: `app-server` (IDE/extension) or `cli` (terminal) |
| **Duration** | Wall-clock duration |
| **Tokens** | Total tokens (input + output + cached) |
Click any row to open a **Session Detail** drawer
showing a turn-by-turn timeline of every event.
### Session Detail Drawer
The drawer displays:
- **Session metadata**: user, model, app version
- **Aggregated stats**: tokens, tool calls, errors,
duration (only populated fields are shown)
- **Turn-by-turn timeline**: each turn is collapsible
and shows individual events (SSE events, WebSocket
events, tool calls). Every event row is expandable
to reveal the full raw JSON payload.
## What Gets Collected
### Metrics
Codex exports the following as OpenTelemetry metrics
(delta temporality):
| Metric | Extra Labels | Description |
|--------|--------------|-------------|
| `codex_turn_token_usage` | `token_type` | Token count by type (input, output, cached, reasoning, total) |
| `codex_thread_started` | none | Conversations started |
| `codex_turn_tool_call` | none | Tool invocations per turn |
| `codex_websocket_request` | `success` | WebSocket API requests |
| `codex_turn_e2e_duration_ms` | none | End-to-end turn latency |
| `codex_turn_ttft_duration_ms` | none | Time to first token |
| `codex_turn_ttfm_duration_ms` | none | Time to first message |
| `codex_websocket_event_duration_ms` | none | WebSocket event processing time |
| `codex_startup_prewarm_duration_ms` | none | Startup prewarm latency |
| `codex_shell_snapshot_duration_ms` | none | Shell snapshot capture latency |
| `codex_turn_network_proxy` | none | Turns routed through the network proxy |
All native metrics carry `model`, `originator`, and
`app_version`. These power the Performance and
Reliability dashboard.
Oodle also derives session-level metrics from the
event logs below: `oodle_logs_codex_token_usage`
(tokens keyed by conversation, user, model, surface,
version, and token type), `oodle_logs_codex_tool_count`
and `oodle_logs_codex_tool_duration_ms` (per-tool
activity with MCP server and success dimensions), and
`oodle_logs_codex_prompt_length_bytes` (prompt sizes).
These power the Usage and Adoption, Token Consumption,
and Tools and Automation dashboards plus the Sessions
tab, and they honor the sidebar User and Surface
filters.
### Events (Logs)
Events are exported via the OpenTelemetry logs
protocol. Each event has an
`attributes.event.name` field:
| Event Type | Key Attributes |
|------------|----------------|
| `codex.conversation_starts` | Conversation ID, model, user email |
| `codex.sse_event` | Model, event kind, token counts, duration |
| `codex.websocket_event` | Model, event kind, duration, success |
| `codex.websocket_request` | Model, duration, success |
| `codex.websocket_connect` | Model, duration |
| `codex.user_prompt` | Prompt text, prompt length |
| `codex.tool_decision` | Tool name, decision, source |
| `codex.tool_result` | Tool name, success, duration, error |
Events are grouped by `conversation.id` to reconstruct
the turn-by-turn timeline on the Sessions tab.
## Further Reading
- [Codex Observability Documentation](https://developers.openai.com/codex/config-advanced#observability-and-telemetry)
- [OpenTelemetry Integration](/integrations/logs/otel)
---
## Factory Droid
Factory's Droid CLI has native OpenTelemetry support.
Once enabled, Oodle collects metrics so you can track
adoption, engineering impact, tool activity, and the
repositories Droid works in across your organization.
## Getting Started
### 1. Enable Telemetry
The fastest way is to use the integration tile in the
Oodle UI:
1. Navigate to **Settings → Integrations**
2. Open the **AI Observability** section
3. Click the **Factory Droid** tile
4. Select an API key and follow the steps shown
Alternatively, set two environment variables wherever
Droid runs (shell profile, CI environment, or
MDM-managed configuration):
```bash
# Droid appends /v1/metrics to the endpoint
export OTEL_TELEMETRY_ENDPOINT=https://
export OTEL_TELEMETRY_HEADERS="X-API-KEY=,X-OODLE-INSTANCE="
```
Replace ``, ``, and
`` with values from the integration tile.
Metrics export every 60 seconds while sessions are
active, and flow to both Factory and Oodle
simultaneously. A failure to reach Oodle never affects
Droid itself.
### 2. Roll Out to Your Team
Add the two environment variables to your dotfiles
repository, developer machine images, or configuration
management tool.
### 3. Verify Data
Once telemetry starts flowing, navigate to
**AI Cost Management → Factory Droid** in the Oodle
sidebar.
## Analysis Page
The Factory Droid page is organized around the
questions engineering leadership asks, with a tab per
use case.
### Adoption
Is Factory getting adopted, and by whom?
| Panel | Description |
|-------|-------------|
| **Active Users / Sessions** | Distinct users and sessions with Droid activity |
| **Logins and Slash Commands** | Engagement signals over the selected range |
| **Activity by Model** | Which models drive Droid usage |
| **Model Mix / Autonomy Mode / Environment** | How Droid is run: model split, autonomy levels, interactive vs other environments |
| **Version Adoption** | How quickly the team upgrades the Droid CLI |
| **Top Users by Activity** | The power users driving adoption, sorted and paginated |
### Engineering Impact
What is Droid actually producing?
| Panel | Description |
|-------|-------------|
| **Files Modified / Lines Modified** | Code change volume attributable to Droid |
| **Commits and PRs Created** | How much of Droid's work ships |
| **Files Read vs Files Modified** | Exploration vs change balance |
| **By Extension** | Which parts of the stack Droid touches most |
| **Per-User Leaderboards** | Lines, commits, and PRs by developer |
### Tools & Automation
How does Droid work, and where does it struggle?
| Panel | Description |
|-------|-------------|
| **Tool Invocations and Success Rate** | Overall tool call volume and health |
| **Execution Time (p50 / p95)** | Tool latency trends and per-tool p95 |
| **Tool Failures** | Which tools fail most |
| **MCP Tools / Skills / Hooks** | Automation surface usage: MCP calls by server, skill invocations, hook activity |
| **Installed Skills Inventory** | Which skills are installed, and where |
### Repo Insights
Where across your codebase estate does Droid operate,
and how healthy are those repos? Built from the
`droid_repo_metadata` metric that Droid emits for every
session:
| Panel | Description |
|-------|-------------|
| **Repos Touched / Sessions per Repo** | Coverage of your repository estate |
| **Repos with Tests / CI / Linters / Docs** | Guardrail coverage of the repos Droid works in |
| **Primary Language Distribution** | Language mix of touched repos |
| **Repo Profile** | Per-repo size, languages, contributors, and commit velocity |
| **Repo Guardrails** | Test frameworks, CI/CD, linters, security, and observability tooling per repo |
:::note
Repositories are identified by a privacy-preserving
hash of the remote URL, so repository names and URLs
never leave the developer's machine.
:::
### Sessions
The Sessions tab lists individual Droid sessions built
entirely from metrics:
| Column | Description |
|--------|-------------|
| **Start Time** | When the session began |
| **Session** | Droid session ID |
| **User** | Email of the developer |
| **Duration** | Wall-clock duration |
| **# Tools** | Tool invocations in the session |
| **Files Read** | Files Droid read |
| **Commits** | Commits created |
| **# PRs** | Pull requests created |
## What Gets Collected
Droid exports the following OpenTelemetry metrics with
delta temporality. Every data point carries
`user.email`, `user.id`, `session.id`, `model_id`,
`organization.id`, and `service.version`, so all views
can be filtered by user, model, and version.
| Metric | Extra Labels | Description |
|--------|--------------|-------------|
| `droid.tool.invocations` | `tool_name`, `tool_category`, `tool_runtime`, `tool_succeeded`, `environment_type`, `session_autonomy_mode` | Tool invocations |
| `droid.tool.execution_time` | same as invocations | Tool execution time histogram (ms) |
| `droid.code.files_read` | `file_extension` | Files read |
| `droid.code.files_modified` | `file_extension` | Files modified |
| `droid.code.lines_modified` | | Lines of code modified |
| `droid.git.commits` | | Commits created |
| `droid.git.pull_requests` | | Pull requests created |
| `droid.mcp.tool_invocations` | `mcp_server`, `tool_name` | MCP tool invocations |
| `droid.skill.invocations` | `skill_name` | Skill invocations |
| `droid.hook.invocations` | | Hook invocations |
| `droid.slash_command.invocations` | `slash_command_name` | Slash command usage |
| `droid.auth.login_success` | `auth_login_method` | Successful logins |
| `droid.repo.metadata` | `repo_*` profile labels | Per-session repository profile snapshot (language, size, tests, CI, linters, contributors, commit velocity) |
## Further Reading
- [Factory Telemetry Documentation](https://docs.factory.ai/enterprise/telemetry-export)
- [OpenTelemetry Integration](/integrations/logs/otel)
---
## AI Cost Management
Oodle's AI Cost Management gives engineering teams
visibility into how AI coding agents are used across
the organization. Track token usage, costs, session
activity, tool calls, and more, all in one place.
## Why Monitor AI Agents?
AI coding agents are powerful but opaque. Without
observability you cannot answer questions like:
- **How much are we spending** on AI-assisted coding?
- **Which models** are being used, and by whom?
- **How many sessions** run per day or week?
- **Are tool calls succeeding**, or are agents hitting
errors?
- **What prompts** are developers sending to the agent?
Oodle collects telemetry exported by AI agents via
OpenTelemetry and surfaces it through a dedicated
analysis page with pre-built dashboards and a
session-level event timeline.
## Supported Agents
| Agent | Metrics | Events / Logs | Status |
|-------|---------|---------------|--------|
| [**Claude**](./claude-code) | Token usage, cost, sessions, active time, lines of code, commits, PRs | User prompts, API requests, tool results, tool decisions | GA |
| [**OpenAI Codex**](./codex) | Token usage, conversations, tool calls, WebSocket requests, performance timings | SSE events, WebSocket events, conversation starts, tool decisions, tool results | GA |
| [**Factory Droid**](./factory) | Tool invocations, files read/modified, lines of code, commits, PRs, skills, repo profiles | Metrics only | GA |
## Getting Started
1. Navigate to **Settings → Integrations**
2. Open the **AI Observability** section
3. Click the tile for the agent you want to set up
4. Select an API key and follow the steps shown
See the agent-specific pages for detailed setup
instructions:
- [Claude Setup](./claude-code#getting-started)
- [OpenAI Codex Setup](./codex#getting-started)
- [Factory Droid Setup](./factory#getting-started)
## Analysis Page
The AI Agent Analysis page is available in the sidebar
under **AI Cost Management**. It has a set of dashboard
tabs tailored to each agent, plus a **Sessions** tab.
Use the **agent dropdown** at the top of the sidebar to
switch between Claude, Codex, and Factory Droid.
### Dashboard Tabs
Each agent gets pre-built dashboards organized around
the questions leadership asks: adoption, cost and ROI,
engineering impact, and tool activity. For example,
Claude has Usage and Adoption, Cost and ROI,
Engineering Impact, Tools and Automation, and Efficiency
tabs; see each agent's page for its tab set.
### Sessions Tab
Shows a table of individual sessions with key stats.
Click any row to open a **Session Detail** drawer
showing a turn-by-turn timeline of every event.
### Sidebar Filters
A sidebar on the left lets you filter data by:
- **Time Range**: Grafana-style time range picker
- **Event Type**: filter events by type
(sessions tab only)
- **Status**: OK or HasError (sessions tab only)
- **Model**: filter by LLM model
- **User**: filter by developer email
- **App Version**: filter by agent version
- **Terminal**: filter by terminal type
## Best Practices
- **Start with a single user** to verify telemetry is
flowing before rolling out org-wide.
- **Enable prompt logging** to get full prompt text in
the event timeline, which is useful for understanding
usage patterns.
- **Set up alerts** on cost or token metrics to catch
unexpected spending spikes.
- **Use sidebar filters** to drill down by user or
model when investigating specific sessions.
---
## Skills & Runbooks
Skills and runbooks add domain expertise to Oodle's AI assistant. They
capture your team's debugging workflows, investigation playbooks, and
operational procedures so the AI can follow them automatically.
When you ask a question in [AI Canvas](/ai-canvas) or through
[MCP](/integrations/mcp), Oodle detects relevant skills and applies
them to guide its investigation. You can also attach skills directly
to [monitors](/alerts/monitors) so the AI follows a specific runbook
when that alert fires.
{/*
*/}
## Concepts
### Skills
Skills are structured investigation workflows. Each skill has a
**name**, a **description** that tells the AI when to use it, and
a set of **steps** written in natural language - similar to an
`AGENTS.md` file. The AI follows these steps when the skill is
triggered.
### Runbooks
Runbooks are attached to specific monitors and guide the AI when
that alert fires. Unlike skills, runbooks are **created
automatically** - Oodle learns from how your team investigates
incidents and generates a runbook to handle future instances
of the same alert. You can review, edit, or disable
auto-generated runbooks at any time.
## Creating a Skill
1. Navigate to **AI Center > Skills** in the sidebar.
2. Click **+ New Skill** in the top-right corner.
3. Fill in the following fields:
| Field | Description |
|-------|-------------|
| **Name** | A short identifier for the skill (up to 100 characters). |
| **Description** | Tells the AI when to use this skill. Be specific - for example, *"Use when the user asks whether an issue affects one customer or multiple customers and wants cross-cluster comparison."* |
| **Steps** | The investigation procedure the AI should follow. Supports Markdown. Use the **Write**, **Preview**, and **Upload file** tabs to author the content. |
4. Click **Save**.
### Writing Effective Steps
Write steps the way you would brief a new on-call engineer. Include:
- What to check first and in what order.
- Which metrics, logs, or traces to query.
- How to interpret the results.
- What output format to produce (verdict, comparison table,
confidence level).
**Example - API query service fan-out investigation:**
> **Name:** `api-query-fanout-investigation`
>
> **Description:** When investigating api-server query errors,
> look for massive fan-out across shards and HTTP 400 error
> responses returning unexpected content.
```markdown
When api-server query errors spike, investigate the
query-worker container logs.
1. Look for 'StatusCode: 0, canceled, context canceled'
warnings - this indicates a massive fan-out across
shards (e.g., data-partition-{0-9}) where queries are
being canceled due to timeouts.
2. Check for 'StatusCode: 400' errors returning HTML/XML
instead of expected JSON. This means an upstream
dependency is returning an error page, causing
deserialization failures in the query path.
3. Verify if the failing queries are missing required
parameters like 'start' or 'end' timestamps. Missing
parameters cause the upstream to reject the request
with a 400.
4. If fan-out is confirmed, check whether the query scope
is too broad (too many shards) and whether adding
tighter label filters would reduce the blast radius.
```
## Attaching Skills to Monitors
You can attach a skill or runbook to a monitor so the AI
references it when investigating that alert:
1. Open the monitor you want to configure.
2. In the monitor settings, select a skill under the
**Runbook** field.
3. Save the monitor.
When this monitor fires and you ask the AI about it, the
attached runbook is loaded automatically.
## Skill Sources
Skills can come from two sources, visible in the **Source**
filter on the Skills & Runbooks page:
| Source | Description |
|--------|-------------|
| **User** | Created manually by your team. |
| **Auto-learned** | Generated by Oodle based on observed patterns and past investigations. You can review, edit, or disable these. |
## Enabling and Disabling Skills
Each skill has an **Enabled** toggle. Disabled skills are not
loaded by the AI during conversations. Use this to temporarily
remove a skill without deleting it.
## Using Skills
### In AI Canvas
Skills are loaded on demand. When you ask a question in
AI Canvas, the AI matches your query against skill
descriptions and applies relevant skills automatically.
You can also reference a skill by name in your prompt.
### Via MCP
Skills are exposed through the Oodle MCP server. Any AI
agent connected to Oodle's MCP - such as
[Claude Code](/integrations/mcp/claude) or
[Cursor](/integrations/mcp/cursor) - can discover and
invoke skills during a conversation.
### With Alerts
When you ask the AI about a firing alert that has an
attached runbook, the AI follows the runbook steps to
investigate and report findings.
## Managing Skills
From the Skills & Runbooks page you can:
- **Search** skills using the search bar.
- **Filter** by type (Skill / Runbook), source
(Auto-learned / User), or status (Enabled).
- **Edit** a skill by clicking on it to open the edit
dialog.
- **Clone** an existing skill to create a variation.
- **Delete** a skill from the edit dialog.
---
## Terraform for alerts
Configure all your alerts, notifiers and notification policies through terraform.
## Advantages of managing alerts as code
* They are version-controlled, making changes and rollbacks more transparent and trackable.
* This approach allows you to standardize alert configurations across environments, eliminating manual
errors and drift between staging and production. You can apply the same alert configuration across
all your deployments / environments.
* Terraform's declarative nature also makes it easy to spin up or tear down alerts alongside your
infrastructure, ensuring your monitoring evolves with your stack.
* Integrating alert management into your CI/CD pipeline means faster, more reliable deployments,
as your alerts will always align with the infrastructure they monitor.
## How
:::info
Use oodle's [Terraform Provider](https://registry.terraform.io/providers/oodle-ai/oodle/latest/docs)
:::
### Prerequisites
To setup Oodle Alerts via Terraform files, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
### Example usage
```hcl
terraform {
required_providers {
oodle = {
source = "registry.terraform.io/oodle-ai/oodle"
}
}
}
# These can also be set as environment variables:
# export OODLE_DEPLOYMENT=https://us1.oodle.ai/
# export OODLE_INSTANCE="my-instance"
# export OODLE_API_KEY="my-api-key"
provider "oodle" {
deployment_url = "https://us1.oodle.ai/"
instance = "my-instance"
api_key = "my-api-key"
}
# Example usage of notifier, notification policy and monitor.
# Refer to https://registry.terraform.io/providers/oodle-ai/oodle/latest/docs/resources/notifier
# for schema documentation.
resource "oodle_notifier" "notifier_test1" {
name = "terraform_test_notifier"
type = "pagerduty"
pagerduty_config = {
service_key = "foo"
send_resolved = true
}
}
# Refer to https://registry.terraform.io/providers/oodle-ai/oodle/latest/docs/resources/notification_policy
# for schema documentation.
resource "oodle_notification_policy" "test1" {
name = "terraform_test_policy"
notifiers = {
critical = [oodle_notifier.notifier_test1.id]
}
}
# Refer to https://registry.terraform.io/providers/oodle-ai/oodle/latest/docs/resources/monitor
# for schema documentation.
resource "oodle_monitor" "test1" {
name = "terraform_test"
promql_query = "sum(rate(oober_food_delivery_revenue_usd[3m]))"
conditions = {
critical = {
value = 1210000
operation = ">"
for = "3m"
alert_on_no_data = true
}
}
notification_policy_id = oodle_notification_policy.test1.id
}
```
Apply terraform configuration
```bash
terraform init
terraform apply
```
---
## Alerts
export const FilteredDocCardList = () => {
const category = useCurrentSidebarCategory();
const filteredItems = category.items.filter(item => item.label !== 'Alerts');
return ;
};
## Entities
Oodle's alerting system allows you to monitor your metrics and receive notifications
when certain conditions are met. There are three entities that make up the alerting system:
- Monitors
- Notification Policies
- Notifiers
Additionally, you can create composite monitors that combine multiple monitor conditions.
Alerts functionality is available in the Oodle UI under the Bell icon in the sidebar navigation.
## Monitors
Monitors are the core of the alerting system. They define:
- A PromQL query to evaluate
- Conditions for triggering alerts (`Warning` and `Critical` thresholds)
- [Notification policy](#notification-policies) to be used for routing alerts
- Labels allow specifying a set of additional labels to be attached to the alert.
- Annotations specifies a set of informational labels that can be used to store
longer additional information such as alert descriptions or runbook links.
Labels and annotations can be templated in a similar manner as documented
in [Prometheus alertmanager](https://prometheus.io/docs/prometheus/latest/configuration/template_examples/).
Monitors are periodically evaluated and alerts are triggered when threshold
conditions are met.
See the [Monitors documentation](./monitors.mdx) for detailed configuration instructions.
## Notifiers
[Notifiers](./notifiers.mdx) are the endpoints where alerts are sent. Oodle supports several types of notifiers:
- [Slack](./notifiers.mdx#slack)
- [PagerDuty](./notifiers.mdx#pagerduty)
- [Opsgenie](./notifiers.mdx#opsgenie)
- Webhook
- [Google Chat](./notifiers.mdx#google-chat)
- [Microsoft Teams](./notifiers.mdx#microsoft-teams)
See the [Notifiers documentation](./notifiers.mdx) for detailed configuration instructions for each notifier type.
## Notification Policies
Notification policies determine how alerts are routed to different notifiers. Policies allow you to:
- Define separate notifiers for `Warning` and `Critical` alerts
- Create global policies that apply to all monitors in addition to the specific policies attached
to the monitors.
See the [Notification Policies documentation](./notification-policies.mdx) for detailed configuration instructions.
## Composite Monitors
Composite monitors allow you to create monitors based on combinations of multiple monitors. Using the `ALERTS` metric, you can implement complex logic such as:
- Alert only when multiple conditions are met simultaneously (AND logic)
- Alert when any of several conditions are met (OR logic)
- Alert when a threshold number of monitors are firing
For example, you can create a composite monitor that triggers only when both a database monitor and an API service monitor are firing, helping you identify dependent service failures.
See the [Composite Monitors documentation](./composite-monitors.mdx) for detailed configuration instructions and examples.
---
## Composite Monitors
Composite monitors allow you to create monitors based on combinations of other monitors.
This is useful when you need to alert on complex conditions that depend on multiple monitors.
## What are Composite Monitors?
Composite monitors use the `ALERTS` metric, which is automatically generated by Oodle for all active alerts.
By querying this metric, you can create monitors that trigger based on the state of other monitors.
## Creating a Composite Monitor
To create a composite monitor, you'll use the `ALERTS` metric in your PromQL query.
Each alert has several labels that you can filter on:
- `_oodle_monitor_id`: The unique ID of the monitor that generated the alert
- `alertstate`: The current state of the alert (`firing` or `pending`)
- `alertname`: The name of the alert
- `_oodle_severity`: The severity level (`critical`, `warn` or `no_data`)
### Basic Example: AND Logic
To alert when **both** Monitor A and Monitor B are firing:
```promql
(
count(ALERTS{
_oodle_monitor_id="",
alertstate="firing"
})
)
and
(
count(ALERTS{
_oodle_monitor_id="",
alertstate="firing"
})
)
```
This composite monitor will only trigger when both specified monitors are actively firing.
Set the condition to trigger when the result is greater than 0, indicating both alerts are firing.
### OR Logic Example
To alert when **either** Monitor A or Monitor B is firing:
```promql
count(ALERTS{
_oodle_monitor_id=~"|",
alertstate="firing"
})
```
### Severity-Based Composite Monitor
To alert when any critical alert from a specific set of monitors is firing:
```promql
count(ALERTS{
_oodle_monitor_id=~"monitor-id-1|monitor-id-2",
alertstate="firing",
_oodle_severity="critical"
})
```
## Finding Monitor IDs
To create composite monitors, you'll need the monitor IDs of the monitors you want to combine. You can find a monitor's ID:
1. Navigate to the monitor in the Oodle UI
2. The monitor ID is displayed in the URL params
3. You can also use labels to identify monitors instead of IDs
---
## Migrations
Oodle supports importing alerts from various platforms to make your migration
seamless. Choose the platform you're migrating from:
## Alertmanager / Prometheus
Import your Prometheus alert rules and Alertmanager notification routes into
Oodle.
**Methods available:**
- Upload `prometheus.yaml` and `alertmanager.yaml` configuration files directly
- Connect to running Prometheus/Alertmanager instances and scrape alerts
👉 [Import Alertmanager Alerts](/integrations/metrics/prometheus/alertmanager-alerts)
## Grafana
Import your existing Grafana Dashboards, Alerts, and Datasources into Oodle
automatically using a Service Account Token.
**What gets imported:**
- Dashboards with panels, queries, and folder structure
- Alert rules configured in Grafana
- Datasource configurations
👉 [Import from Grafana](/integrations/metrics/grafana)
---
## Monitors
Monitors allow you to define metric queries that trigger alerts based on specific conditions.
The system periodically evaluates these queries and generates alerts when the results violate the defined conditions.
## Key Components of a Monitor
### Alert Name
A descriptive name that identifies the monitor. You can use template variables in
the name using [Prometheus templates](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/) syntax.
### Query
A PromQL query that the system evaluates periodically. You can use any valid PromQL query syntax.
### Conditions
Conditions specify the threshold values that trigger alerts. There are two severity levels:
- `Critical`
- `Warning`
You must define at least one condition, and you can set different thresholds for each severity level.
The following operators are available for conditions:
- Equal to
- Not equal to
- Above
- Above or Equal to
- Below
- Below or Equal to
### Message
A customizable message that accompanies alert notifications. You can enhance messages using template variables
with [Prometheus templates](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/) syntax.
For example, with a query like `max by (job) (up)`, you can access the job name in your message using
the template variable `{{ $labels.job }}`.
:::note
Oodle notifications automatically include the following information, so you don't need to add them to your message:
- Alert name
- Severity
- Threshold
- Metric Value (the current value when the alert fires)
- Labels and Annotations
:::
### Notification Policy
Notification policies determine where alert notifications
are sent. You can configure multiple notifiers as
destinations for your alerts.
Each rule specifies which notifiers to trigger and can
optionally be scoped by label conditions and severity.
#### How Routing Rules Work
Notification rules are evaluated **top to bottom** on a
first-match basis. The first rule whose conditions match
the alert fires its notifiers; subsequent rules are
skipped. The last rule without any label conditions acts
as the **fallback** and catches everything that didn't
match an earlier rule.
Rules follow an **If / Else If / Else** pattern:
| Position | Prefix | Description |
|----------|--------|-------------|
| First rule (with conditions) | **If** | Evaluated first |
| Additional rules (with conditions) | **Else If** | Evaluated only when preceding rules don't match |
| Last rule (no conditions) | **Always** / **Else** | Fallback — matches all remaining alerts |
You can drag and drop rules to reorder them.
#### Label-Based Routing
Each rule can include one or more **label conditions** to
match alerts. Conditions within a single rule are combined
with AND logic. Available operators are:
- `is` — exact match
- `is not` — negated match
- `matches regex` — regex match
- `does not match regex` — negated regex match
Label values are auto-suggested from the monitor's query
results, so you can pick from labels that actually appear
in your data.
**Example — Team-based routing:**
Route alerts to the owning team's channel based on a
`team` label:
| Rule | Condition | Notifier |
|------|-----------|----------|
| **If** | `team` is `backend` | Slack `#backend-alerts` |
| **Else If** | `team` is `frontend` | Slack `#frontend-alerts` |
| **Else** | *(none — fallback)* | Slack `#ops-alerts` |
**Example — Environment-based routing:**
Escalate production alerts to your incident management
tool while keeping non-production alerts in Slack:
| Rule | Condition | Notifier |
|------|-----------|----------|
| **If** | `environment` is `production` | OpsGenie |
| **Else** | *(none — fallback)* | Slack `#dev-alerts` |
**Example — Combined conditions:**
Conditions within a rule are AND'd together, so you can
be very specific:
| Rule | Conditions | Notifier |
|------|------------|----------|
| **If** | `team` is `payments` AND `environment` is `production` | PagerDuty |
| **Else If** | `environment` is `production` | OpsGenie |
| **Else** | *(none — fallback)* | Slack `#alerts` |
#### Severity-Based Routing
Within each rule, you can filter by alert severity using
the **for** dropdown. The options are:
| Option | Matches |
|--------|---------|
| **all** | Critical, Warning, and No Data alerts |
| **critical** | Only Critical alerts |
| **warning** | Only Warning alerts |
| **no data** | Only No Data alerts |
When you select a specific severity, you can add
additional severity rows within the same rule to send
different severities to different notifiers. For example,
within a single rule you can send `critical` alerts to
PagerDuty and `warning` alerts to Slack.
#### Notification Policies vs. Direct Notifiers
Each rule supports two action types, selectable via the
dropdown next to the rule:
- **notify on** — Send directly to one or more notifiers
(Slack, OpsGenie, PagerDuty, etc.)
- **trigger policy** — Delegate to a reusable
[Notification Policy](/alerts/notification-policies)
that defines its own routing, grouping, and timing
### Grouping
Grouping controls how notifications are consolidated. You can choose from three options:
- By Monitor: Receive a single notification per monitor (default behavior)
- By Labels: Receive a notification for each unique set of specified labels
- Disabled: Receive a notification for each individual timeseries
For example, consider a monitor with this query that alerts when the `up` metric for any `job` is 0:
```
max by (job, instance) (up{})
```
Here's how different grouping options affect notifications:
- **By Monitor:** You receive one notification that includes all affected jobs and instances
- **By Labels** (with `job` specified): You receive one notification per affected job
- **Disabled:** You receive one notification per affected timeseries (job-instance combination)
### Notification Timings
These parameters control the timing of grouped notifications:
#### Group Wait
Group Wait defines how long to wait before sending the first notification for a new group of alerts.
The default is 30 seconds.
#### Group Interval
Group Interval defines how long to wait before sending a notification for new alerts added to a group that has already sent notifications.
The default is 5 minutes.
#### Repeat Interval
The repeat interval specifies how often notifications are resent for active alerts. For example, if you have an active alert
for the timeseries `up` with `job=foo` and `instance=bar`, setting a repeat interval of 10 minutes means the notification
will be sent every 10 minutes until the alert is resolved.
:::info
For example, with the `up` metric monitor grouped "By Monitor", consider a group wait of 30 seconds and a group interval of 5 minutes:
If two jobs go down within the first 30 seconds, you'll receive one notification covering both jobs. If a third job goes down a minute later,
its notification will be delayed by the 5-minute group interval.
:::
### Labels
Labels add metadata to alerts. You can use [Prometheus templates](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/) syntax
to create dynamic labels.
A common use case is tagging monitors with owner teams to identify responsible teams for each monitor.
### Annotations
Annotations provide additional context for alerts, such as descriptions or runbook links.
Like labels, annotations support [Prometheus templates](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/) syntax.
## Creating a Monitor with AI
The quickest way to create a monitor is to ask the AI Assistant. Open
the sidebar assistant (Cmd + I) or
[AI Canvas](/ai-canvas) and describe the alert you need:
- `Create an alert to notify me when container memory usage is above 90%`
- `Alert me when the error rate on api-server exceeds 5%`
- `Create an alert for container restarts in the last 10 minutes`
The AI will build the PromQL query, set the threshold, and save the
monitor. You can review and adjust it from the alerts page afterwards.
## Creating a Monitor Manually
1. Navigate to **Alerts** (Bell icon in sidebar)
2. Click **New Alert**
3. Configure the following settings:
- **Name:** Enter a descriptive name for your monitor
- **Query:** Specify the PromQL query to evaluate
- **Conditions:** Define the alerting conditions
- **Message:** (Optional) Add a message for alert notifications
- **Notification Policy:** (Optional) Select routing rules for alerts
- **Grouping:** (Optional) Configure how you want to group alerts. Additionally,
you can configure various timing parameters by clicking on the **Clock** icon.
- **Labels:** (Optional) Add metadata labels to the alerts
- **Annotations:** (Optional) Include additional contextual information
4. Click **Save** to create the monitor
---
## Muting
## Overview
You can mute alerts to temporarily silence notifications for specific alerts without deleting or disabling them permanently. This is useful during maintenance windows, known incidents, or when you need to reduce noise.
## When to Use
Mute alerts when:
- Performing scheduled maintenance
- Investigating a known issue
- Testing changes in production
- Dealing with expected high traffic events
- Reducing alert fatigue during incident response
## How to Mute an Alert
1. Go to the Alerts page.
2. Find the alert you want to mute.
3. Click on alert's context menu [•••] and select 'Mute'.
4. Select the duration for muting (options: 1 hour, 6 hours, 12 hours, 1 day, 2 days, infinite, or custom).
5. Optionally add labels for muting.
6. Click 'Add Muting Rule'.
7. To unmute, click on the red trash can.
## How to Unmute an Alert
1. Navigate to the Alerts page.
2. Filter or find muted alerts using the Muted filter in the left sidebar, or sort the 'Muted left' column.
3. Click on a muted alert's Muted indicator in the Muted left column.
4. Click on the red trash can to permanently remove it.
## Muted Alert Behavior
- Muted alerts do not trigger notifications via email, Slack, PagerDuty, or other channels
- Muted alerts still evaluate conditions, and change state to "firing"
- You can view muted alert firing history in the alert timeline
- Muted alerts appear with a 'Muted' badge in the alerts list
- The system automatically unmutes alerts when the mute period expires
## FAQ
### Can I mute multiple alerts at once?
Yes, you can select multiple alerts from the alerts list and mute in-bulk using the bulk actions menu.
### What happens if an alert triggers while muted?
The alert condition is still evaluated, the state of the alert is appropriately updated, but no notifications are sent. You can see these occurrences in the alert history.
### Can I schedule a mute in advance?
Yes, when selecting 'Custom' duration, you can set a future start time for the mute period.
### Are there limits on mute duration?
No, there are no maximum limits on mute duration.
## Related Features
- [Alert Conditions](/alerts/monitors#conditions) - Modify alert thresholds to reduce false positives
- [Notification Policies](/alerts/notification-policies) - Configure which channels receive alert notifications
- [Notifiers](/alerts/notifiers) - Configure notification channels
---
## Notification Policies
Notification policies let you route alerts to different notifiers based on severity levels.
Each policy can be configured with specific notifiers for both `Warning` and `Critical` severity levels.
With notification policies, you can:
- Create team-specific policies to route team alerts to designated notifiers
- Set up deployment-specific policies to route alerts from different deployments (e.g. dev, staging, production) to appropriate notifiers
## Create a Notification Policy
1. Navigate to **Alerts** (Bell icon in sidebar) → click the ellipsis (**...**) → select **Notification Policies**
2. Click the **+** icon.
3. Fill in the following fields:
- Name: A descriptive name for your notification policy
- Critical: (Optional) Select notifiers to receive alerts with `Critical` severity
- Warning: (Optional) Select notifiers to receive alerts with `Warning` severity
- Global: Enable this toggle to apply the policy to all monitors in addition to their individual notification policies.
This is particularly useful when migrating from another system and you want to verify Oodle alerts while maintaining
your existing alert system. By default, this toggle is **OFF**.
4. Click **Save** to create the notification policy
:::note
You can also create notification policies directly from the monitor create/edit page by clicking the **Add Notification** button.
:::
## Use a Notification Policy
After creating a notification policy, you can attach it to any monitor during creation or editing.
Once attached, the monitor's alerts will be automatically routed to the notifiers specified in the policy.
---
## Notifiers
Notifiers are endpoints that receive alert notifications when monitor conditions are triggered.
This guide explains each supported notifier type and provides step-by-step configuration instructions.
## Supported Notifiers
### Slack
To send notifications to Slack, you'll need an incoming webhook URL. There are two methods to create one:
1. Slack Apps
2. Incoming Webhooks (Legacy)
#### Slack Apps
Follow these steps to create an incoming webhook using Slack Apps. For more details, see the
official [Slack documentation on incoming webhooks](https://api.slack.com/messaging/webhooks).
1. Create a Slack App:
- Visit [https://api.slack.com/apps?new_app=1](https://api.slack.com/apps?new_app=1)
- Choose a name for your app
- Select your workspace
- Click **Create App**
2. Enable incoming webhooks:
- Open the [Apps management dashboard](https://api.slack.com/apps)
- Select your app
- Click on Incoming Webhooks
- Toggle **Activate Incoming Webhooks** to ON
3. Create a webhook:
- Click **Add New Webhook to Workspace**
- Select the channel for receiving messages
- Click **Authorize**
You'll receive a webhook URL that looks like this:
```
https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
```
:::note
Webhooks created through Slack Apps can only post to their designated channel. You'll need separate webhooks for different channels.
:::
#### Incoming WebHooks App (Legacy)
The legacy Incoming WebHooks app allows you to create webhooks that can send messages to any channel in your workspace.
1. Install the [Incoming WebHooks](https://slack.com/marketplace/A0F7XDUAZ-incoming-webhooks) app to your workspace
2. Click **Add to Slack**, select a default channel, then click **Add Incoming WebHooks Integration**
:::note
Legacy webhooks can send messages to any channel in your workspace, unlike Slack Apps webhooks which are channel-specific.
:::
#### Configuration
To set up a Slack notifier in Oodle:
1. Navigate to **Alerts** (Bell icon in sidebar) → click the ellipsis (**...**) → select **Notifiers**
2. Click the **+** icon and choose **Slack** from the dropdown
3. Fill in the following:
- Name: A descriptive name for your notifier
- API URL: The URL from Slack
- Channel (optional): Override the default channel. Note: This only works with legacy webhooks
:::note
For Slack Apps webhooks, messages will always go to the channel specified during webhook creation, regardless of the channel setting in Oodle.
:::
4. Click **Save** to create the notifier
### Pagerduty
To send notifications to Pagerduty, you'll need an API Key. Oodle supports following Pagerduty Integrations:
1. Events API V2 (Recommended)
2. Prometheus
##### Events API V2 (Recommended)
[Events API V2](https://developer.pagerduty.com/docs/events-api-v2-overview) is rich Events API in comparison
to Prometheus integration. For example, it allows to set Pagerduty Alert's Severity, attach images etc. which
are not possible in Prometheus integration type.
To get an Events API v2 API Key, follow:
1. Log in to PagerDuty and navigate to **Services** in the top menu bar.
2. Click on the service you are currently using, or create a new service.
3. Click **Integrations** and then **Add another integration**.
4. Select **Events API V2** from the list of integrations.
5. After creating the integration, you will be able to see the API Key for this integration.
##### Prometheus
Prometheus Integration is less richer in comparison to Events API V2 Integration. To get a Prometheus Integration
API Key, follow:
1. Log in to PagerDuty and navigate to **Services** in the top menu bar.
2. Click on the service you are currently using, or create a new service.
3. Click **Integrations** and then **Add another integration**.
4. Select **Prometheus** from the list of integrations.
5. After creating the integration, you will be able to see the API Key for this integration.
#### Configuration
To set up a Pagerduty notifier in Oodle:
1. Navigate to **Alerts** (Bell icon in sidebar) → click the ellipsis (**...**) → select **Notifiers**
2. Click the **+** icon and choose **Pagerduty** from the dropdown
3. Fill in:
- Name: A descriptive name for your notifier
- Integration: Choose `Events API V2` or `Prometheus` based on the Integration you have configured on Pagerduty.
- API Key: Your Pagerduty integration API key
4. Click **Save** to create the notifier
When using `Events API V2`, Oodle sets [Pagerduty Alert's Severity](https://support.pagerduty.com/main/docs/dynamic-notifications#severity-and-urgency-mapping)
based on the Condition Severity defined within the monitor:
- Critical condition in Oodle is mapped to `critical` severity in Pagerduty.
- Warning condition in Oodle is mapped to `warning` severity in Pagerduty.
Follow [Configure Dynamic notifications](https://support.pagerduty.com/main/docs/dynamic-notifications#configure-dynamic-notifications)
in Pagerduty to govern alert notifications based on Severity.
### Opsgenie
To send notifications to Opsgenie, you'll need an API key. Here's how to get one:
1. Log in to Opsgenie and navigate to [Integrations](https://app.opsgenie.com/settings/integration/integration-list)
2. Click **Add Integration** and select **Prometheus**
3. Configure the integration:
- Enter a name
- Select the team to receive alerts
- Click **Continue**
4. Click **Turn on Integration** to activate it
5. [Optional] In `Incoming | Rules for creating and processing alerts` section, Click on `Create Alert` and update
`Message` field from `[Prometheus]: {{message}}` to `{{message}}`.
#### Configuration
To set up an Opsgenie notifier in Oodle:
1. Navigate to **Alerts** (Bell icon in sidebar) → click the ellipsis (**...**) → select **Notifiers**
2. Click the **+** icon and choose **Opsgenie** from the dropdown
3. Fill in:
- Name: A descriptive name for your notifier
- API Key: Your Opsgenie integration API key
4. Click **Save** to create the notifier
Oodle sets [Opsgenie's Priority](https://support.atlassian.com/opsgenie/docs/what-is-the-priority-level-of-integration/)
based on the Condition Severity defined within the monitor:
- Critical condition in Oodle is mapped to `P1` priority in Opsgenie.
- Warning condition in Oodle is mapped to `P2` priority in Opsgenie.
### Rootly
To send notifications to Rootly, you'll need a Bearer Token Secret from a Rootly Alertmanager source.
1. In Rootly, navigate to [Alert Sources > New Alertmanager Source](https://rootly.com/account/alert-sources/new?alerts_source%5Bsourceable_type%5D=Alerts%3A%3AAlertmanagerSource)
2. Give the source a name (e.g. "Oodle Alerts")
3. Under **Urgency**, add a condition: `$.alerts[0].labels._oodle_severity` is `warn` sets urgency to **Medium**, otherwise **High**
4. Save and copy the **Bearer Token Secret**
#### Configuration
To set up a Rootly notifier in Oodle:
1. Navigate to **Alerts** (Bell icon in sidebar) → click the ellipsis (**...**) → select **Notifiers**
2. Click the **+** icon and choose **Rootly** from the dropdown
3. Fill in:
- Name: A descriptive name for your notifier
- Bearer Token: The Bearer Token Secret from Rootly
4. Click **Save** to create the notifier
Oodle delivers alerts in Alertmanager webhook format. Rootly uses the `_oodle_severity` label to map alert urgency based on the conditions you configured above.
For more details, see the [Rootly Alertmanager docs](https://docs.rootly.com/integrations/alertmanager).
### Google Chat
To send notifications to Google Chat, you'll need a webhook URL. Get the Webhook URL by following
[Create a webhook for the Google Space](https://developers.google.com/workspace/chat/quickstart/webhooks) for the Google Space you want to send notification to.
When creating the webhook in Google Chat, we recommend:
- **Name**: `Oodle Alerts`
- **Avatar URL**: `https://www.oodle.ai/img/logo_square.png`
This gives alert messages a recognizable name and Oodle logo in Google Chat.
#### Configuration
To set up a Google Chat notifier in Oodle:
1. Navigate to **Alerts** (Bell icon in sidebar) → click the ellipsis (**...**) → select **Notifiers**
2. Click the **+** icon and choose **Google Chat** from the dropdown
3. Fill in:
- Name: A descriptive name for your notifier
- Webhook URL: The Webhook URL you created above.
4. Click **Save** to create the notifier.
### Microsoft Teams
To send notifications to Microsoft Teams, you'll need a webhook URL. Get the Webhook URL by following
[Create an incoming Webhook with Workflows](https://support.microsoft.com/en-gb/office/create-incoming-webhooks-with-workflows-for-microsoft-teams-8ae491c7-0394-4861-ba59-055e33f75498) for the Teams Channel you want to send notifications to.
#### Configuration
To set up a Microsoft Teams notifier in Oodle:
1. Navigate to **Alerts** (Bell icon in sidebar) → click the ellipsis (**...**) → select **Notifiers**
2. Click the **+** icon and choose **Microsoft Teams** from the dropdown
3. Fill in:
- Name: A descriptive name for your notifier
- Webhook URL: The Webhook URL you created above.
4. Click **Save** to create the notifier
## Use a Notifier
Once you have a notifier created, you can use it in a notification policy to route alerts to the notifier.
See the [Notification Policies documentation](./notification-policies.mdx) for more details.
---
## Anomaly Alerting
Anomaly Alerting lets you configure notification policies for
anomalies detected by Oodle's anomaly detection system.
When anomalies are detected in your logs or metrics, alerts are
sent to the notification channels you configure.
## Getting Started
Navigate to **Anomalies → Anomaly Alerting** in the sidebar. The
page shows the available anomaly detection modules and their
status.
## Detectors Table
The page displays a table of all anomaly detectors:
| Column | Description |
|--------|-------------|
| **Name** | Detector name (underscores are displayed as spaces). |
| **Description** | What the detector monitors. |
| **Data Sets** | The data source the detector operates on (e.g., `logs`, `metrics`), shown as badges. |
| **Status** | Toggle switch to enable or disable the detector. |
Clicking an enabled detector row opens a drawer that shows
the alerts generated by that detector, using the same interface
as the [Alerts](/alerts/) page.
## Notification Policy
At the top of the page, the **Notification Policy** label shows
the currently assigned policy. Click it to open a drawer where
you can create or select an existing notification policy.
Notification policies define which channels (Slack, email,
PagerDuty, etc.) receive anomaly alerts.
### Setting a Notification Policy
1. Click the **Notification Policy** value (or the "-" if none
is set).
2. In the drawer, select an existing policy or create a new one.
3. Click **Save and Add Notification Policy**.
You can remove the current policy by clicking the delete icon
next to the policy name.
## Enabling / Disabling Detectors
Toggle the **Status** switch for each detector:
- **Enabled** (blue) — The detector actively monitors for
anomalies and sends alerts through the configured notification
policy. The row becomes clickable to view related alerts.
- **Disabled** (grey) — The detector is paused and does not
generate alerts. The row is not clickable.
:::note
Toggling a detector requires edit permissions. Users without
edit access will see disabled switches.
:::
## How It Works
1. Oodle continuously analyzes incoming logs and metrics using
machine learning models.
2. When a pattern deviates significantly from the learned
baseline, an anomaly is detected.
3. If the corresponding module is enabled and a notification
policy is configured, an alert is sent.
4. Detected anomalies are also visible in the
[Anomaly Detection](/anomaly-detection/) dashboard.
## Best Practices
- **Enable both modules** for comprehensive coverage across logs
and metrics.
- **Set up a dedicated notification channel** for anomaly alerts
to avoid mixing them with other alert types.
- **Review anomalies regularly** on the Anomaly Detection
dashboard, even if alerts are configured — not all anomalies
may be severe enough to trigger notifications.
- **Tune notification policies** to avoid alert fatigue. Start
broad and narrow down as you learn which anomalies are
actionable.
## Related Pages
- [Anomaly Detection](/anomaly-detection/) — View and
investigate detected anomalies.
- [Alerts](/alerts/) — Configure alert rules and notification
policies.
---
## Anomaly Detection
Oodle automatically identifies unusual patterns in your metrics and logs,
helping you catch issues before they impact your users.
## Overview
Anomaly detection helps you:
- **Detect Issues Early**: Automatically identify unusual patterns before they
become critical problems
- **Reduce Alert Noise**: Focus on genuine anomalies by blocking repetitive or
irrelevant patterns
- **Compare Historical Patterns**: Understand how current behavior differs from
your historical baselines
- **Prioritize Issues**: Anomalies are ranked by severity to help you focus on
what matters most
## How It Works
Oodle continuously monitors your infrastructure and applications, automatically
detecting:
- **Metric Anomalies**: Unusual behavior in your infrastructure and application
metrics
- **Log Anomalies**: New or unusual patterns in your error logs
When anomalies are detected:
- They appear on the Anomalies page for you to review
- You can set up alerts to be notified immediately
- You can block repetitive or expected patterns to reduce noise
## Types of Anomalies
### Metric Anomalies
Oodle detects unusual patterns in your infrastructure and application metrics,
including:
- **CPU Throttle**: Services experiencing CPU throttling
- **High Node CPU**: Nodes with unusually high CPU usage
- **Service Communication Errors**: Communication issues between services
- **API Errors**: High error rates in your APIs
- **Container Restarts**: Containers restarting unexpectedly
- **High Memory**: Services with excessive memory usage
- **Pod Phase Issues**: Pods in problematic states (e.g., CrashLoopBackOff)
### Log Anomalies
Oodle identifies unusual patterns in your logs by detecting:
- **New Patterns**: Error messages that haven't appeared before
- **Spikes**: Sudden increases in existing error patterns
- **High Volume**: Recurring errors with unusually high occurrence rates
Each anomaly shows whether it's a new issue or a recurring problem, helping you
prioritize your response.
## Using the Anomalies Page
The Anomalies page (`/anomalies`) provides three main tabs:
### Anomalies Tab
View all detected anomalies with comprehensive filtering and details:
- **Filters**: Filter by type (metrics/logs), subtype, cluster, namespace, or
service
- **Time Range**: Select the time window to view anomalies
- **Details**: Click on any anomaly to see:
- Trend charts showing the anomaly over time
- Related logs (for log anomalies)
- Metric queries (for metric anomalies)
- Labels and metadata
### Detectors Tab
Control which types of anomalies to detect:
- Toggle individual detector types on/off
- Changes save automatically
- Useful for temporarily disabling noisy or irrelevant detectors
### Spam Tab
Manage blocked anomalies to reduce noise:
- **Search**: Search through blocked anomalies to find specific items
- **View Blocked Items**: See all anomalies you've blocked
- **Unblock**: Remove items from the blocklist to allow them to appear again
When you block an anomaly, it won't trigger alerts or appear in your anomaly
feed, keeping your view focused on what matters.
## FAQ
### How does Oodle detect log anomalies?
Oodle doesn't evaluate individual log lines. Instead it
works at the **pattern** level:
1. **Filter to errors** — only error-level logs are
considered. Non-error logs are never evaluated.
2. **Group by pattern** — error logs are bucketed by
**cluster**, **container**, and **fingerprint** (a
pattern identifier for structurally similar messages).
3. **Compare against 24 hours ago** — the total count
for each group in the last 1-hour window is compared
against the same 1-hour window 24 hours ago.
4. **Apply thresholds** — a group is flagged when:
- It has **more than 4 occurrences** in the current
hour, **and**
- It meets one of these conditions:
- **New Pattern** — the group didn't exist in the
baseline window
- **Spike** — the count has at least doubled
compared to 24 hours ago
- **Volume** — a recurring error with notable
count that hasn't doubled
### How does Oodle detect metric anomalies?
Oodle runs predefined queries against your infrastructure
and application metrics every 5 minutes:
1. **Query current and historical values** — each query
produces a single value for the last 1-hour window
and a comparison value from the same window 24 hours
ago.
2. **Compare values** — an anomaly is flagged when:
- The current value shows a **significant percentage
increase** over the historical value (used for CPU
throttling, node CPU, memory usage, API error
rates, and service communication errors), or
- The current value is **non-zero** (used for
container restarts and pod phase issues, where any
occurrence is notable).
### What time window is used to detect anomalies?
Both log and metric anomaly detection compare two
one-hour windows:
- **Current window**: the last hour
(`now − 1h` to `now`)
- **Baseline window**: the same hour one day ago
(`now − 25h` to `now − 24h`)
Using the same time-of-day accounts for diurnal traffic
patterns. The checks run every 5 minutes, so anomalies
are re-evaluated with fresh data on each cycle.
### How long after a fix before an anomaly disappears?
Anomalies are not stored as persistent records — they are
re-computed from live data every 5 minutes. Once a fix is
deployed:
1. The one-hour sliding window needs to clear of pre-fix
data. In the worst case this takes **up to one hour**
after the fix.
2. On the next 5-minute check after the window clears, no
new data point is emitted for that anomaly.
3. The anomaly's trend chart in the UI drops to zero and
it effectively disappears from the Anomalies page.
This means it can take **up to 1 hour** after a fix
is deployed for the anomaly to fully disappear.
### Can anomaly detection be customized?
Yes — Oodle's anomaly detection is customizable. Reach
out to us to tailor detection rules based on your
specific data patterns and operational needs.
## Best Practices
1. **Review Regularly**: Check the Anomalies page
regularly to catch emerging issues early
2. **Block Noise**: Use the blocking feature to suppress
known non-issues or expected patterns
3. **Create Targeted Alerts**: Set up alerts for specific
anomaly types that are most critical to your operations
4. **Investigate Recurring Anomalies**: Pay attention to
recurring anomalies, as they may indicate persistent
underlying issues
5. **Use Filters**: Filter by cluster, namespace, and
service to focus on specific areas of your
infrastructure
---
## Dashboards
Dashboards are a way to query, visualize, and organize your observability data
to gain deeper insights at a glance. Oodle provides dashboards for both metrics
and logs.
## Metrics Dashboards
Build dashboards using PromQL to visualize your metrics data. Create panels
with time series graphs, gauges, stat panels, and more.
- **Query with PromQL** - Use the query builder or write queries directly
- **Dashboard variables** - Create dynamic, reusable dashboards
- **Infrastructure as code** - Manage dashboards via Terraform
[Explore Metrics Dashboards →](/dashboards/metrics)
## Logs Dashboards
Create dashboards from your log data to monitor application health, track
errors, and analyze patterns. Build visualizations and combine them into
comprehensive monitoring views.
- **Multiple chart types** - Line, bar, area, pie, heat map, and more
- **Aggregations** - Count, sum, average, and unique count metrics
- **Flexible grouping** - Date histograms, terms, filters, and ranges
[Explore Logs Dashboards →](/dashboards/logs/log-dashboards)
## Explore Further
---
## Dashboards(Logs)
Dashboards combine multiple visualizations into a single view, giving you a
comprehensive overview of your log data. Build dashboards to monitor
application health, track errors, and analyze user behavior.
:::tip Try it Live
Explore a working example dashboard in the
[Oodle Playground](https://play.oodle.ai/logs/app/dashboards)
to see a multi-panel dashboard.
:::
## Terminology
| Term | Description |
|------|-------------|
| **Dashboard** | A collection of panels arranged on a canvas that provides a unified view of your data |
| **Panel** | A single visualization displayed on a dashboard (also called a visualization) |
| **Visualization** | A chart, graph, metric, or table that represents your log data visually |
| **Index Pattern** | The data source that defines which logs are available for visualizations |
## Creating a New Dashboard
1. Navigate to the **Dashboards** section from the sidebar
2. Click **Create Dashboard**
3. You'll see an empty dashboard canvas ready for panels
## Adding Visualizations
### Add Existing Visualization
1. Click **Add** in the toolbar
2. Select **Add from library**
3. Browse or search for saved visualizations
4. Click on a visualization to add it to the dashboard
### Create New Visualization
1. Click **Add** in the toolbar
2. Select **Create new**
3. Choose a visualization type
4. Configure and save the visualization
5. It will automatically be added to your dashboard
## Dashboard Controls
### Time Filter
The time picker in the top navigation controls the time range for all panels.
You can:
- Select predefined ranges (Last 15 minutes, Last 24 hours, etc.)
- Set custom absolute time ranges
- Use relative time expressions
### Query Bar
Enter a query in the search bar to filter data across all panels:
```
container_name:frontend AND log_level:error
```
### Filters
Click **Add filter** to create field-based filters that apply to all panels.
Filters appear as pills below the query bar and can be:
- Toggled on/off
- Pinned across dashboards
- Inverted (exclude matching documents)
### Refresh
- **Manual refresh**: Click the refresh button
- **Auto-refresh**: Set an interval (e.g., every 30 seconds) for live
monitoring
## Panel Options
Click the gear icon on any panel to access:
| Option | Description |
|--------|-------------|
| **Edit visualization** | Modify the underlying visualization |
| **Clone panel** | Duplicate the panel on this dashboard |
| **Create drilldown** | Link to another dashboard or URL |
| **Full screen** | Expand the panel to full screen |
| **Replace panel** | Swap with a different visualization |
| **Delete from dashboard** | Remove the panel |
## Saving Dashboards
1. Click **Save** in the toolbar
2. Enter a dashboard title
3. Optionally add a description
4. Choose whether to save the current time filter with the dashboard
5. Click **Save**
### Save Options
- **Save as new dashboard**: Creates a copy with a new name
- **Store time with dashboard**: Opens dashboard with the saved time range
instead of the default
## Next Steps
- Learn about [creating visualizations](/dashboards/logs/visualizations) to
build custom panels for your dashboards
- Explore the [logs interface](/logs-visualization) for ad-hoc log analysis
---
## Visualizations
Visualizations are the building blocks of [dashboards](/dashboards/logs/log-dashboards).
They transform your log data into charts, graphs, and other visual formats to
identify trends, track metrics, and analyze patterns.
You can create visualizations directly from the **Visualize** section, or create
them while building a dashboard by selecting **Create new** when adding a panel.
:::tip Try it Live
Explore working examples in the
[Oodle Playground](https://play.oodle.ai/logs/app/visualize)
to see visualizations in action.
:::
## Getting Started
To create a new visualization:
1. Navigate to the **Visualize** section from the sidebar
2. Click **Create visualization**
3. Select a visualization type
4. Choose your index pattern (your log data source)
5. Configure metrics and buckets
6. Save your visualization
## Core Concepts
### Metrics
Metrics determine **what** you're measuring. Common metric types include:
| Metric | Description | Example Use Case |
|--------|-------------|------------------|
| **Count** | Number of documents matching the query | Total logs per minute |
| **Sum** | Total of a numeric field | Total bytes transferred |
| **Average** | Mean value of a numeric field | Average response time |
| **Min/Max** | Minimum or maximum value | Peak memory usage |
| **Unique Count** | Distinct values of a field | Number of unique users |
### Buckets
Buckets determine **how** your data is grouped. Common bucket types include:
| Bucket | Description | Example Use Case |
|--------|-------------|------------------|
| **Date Histogram** | Groups data by time intervals | Logs per hour |
| **Terms** | Groups by field values | Logs per service |
| **Filters** | Groups by custom queries | Error vs Success logs |
| **Range** | Groups numeric values into ranges | Response time buckets |
| **Histogram** | Groups by numeric intervals | Request size distribution |
## Visualization Editor
The visualization editor lets you build charts by configuring filters, time
ranges, and aggregations.
| # | Component | Description |
|---|-----------|-------------|
| 1 | **Filter** | Filter logs to include only data relevant to your visualization. Add multiple filters to narrow down results. |
| 2 | **Time Range** | Set the time window for your data. Use quick selections like "Last 1 day" or define custom ranges. |
| 3 | **Metrics (Y-Axis)** | Choose the aggregation for your Y-axis values. Options include Count, Sum, Average, Min, Max, and more. Scroll to see all available aggregations. |
| 4 | **Buckets (X-Axis)** | Define how data is grouped on the X-axis. Use Date Histogram for time-based charts or Terms for categorical grouping. |
| 5 | **Split Series** | Add a sub-aggregation to break down data into multiple series. Select Terms to split by field values. |
| 6 | **Sub-aggregation Field** | Choose the field to split your series by (e.g., `container_name`, `log_level`, `service`). |
## Visualization Types
### Line Chart
Line charts display data points connected over time, ideal for tracking trends
and identifying patterns in time-series data.
**Best for:**
- Tracking metrics over time (transactions, requests, errors)
- Identifying trends and anomalies
- Comparing before/after changes
**Configuration:**
1. **Metric**: Choose Count (or Sum/Average for numeric fields)
2. **X-Axis**: Add a Date Histogram bucket on your timestamp field
3. **Split Series** (optional): Add a Terms or Filters bucket to compare
multiple series
**Example: Successful Transactions Over Time**
- Metric: Count
- X-Axis: Date Histogram on `timestamp` with auto interval
- Split Series: Filters with query `message:Transaction complete.`
→ [View example in Playground](https://play.oodle.ai/logs/app/visualize#/edit/4ef9c1a0-edc0-11f0-b051-df37cf89c1fe)
---
### Bar Chart
Bar charts compare categorical data across groups or time periods. Stacked bars
show the composition of each group.
**Best for:**
- Comparing volumes across categories
- Showing composition (stacked bars)
- Analyzing distribution by time with breakdowns
**Configuration:**
1. **Metric**: Count or Sum
2. **X-Axis**: Date Histogram for time-series, or Terms for categories
3. **Split Series**: Terms bucket on a field (e.g., `container_name`,
`log_level`) to stack by category
**Example: Cart Operations Over Time**
- Metric: Count
- X-Axis: Date Histogram on `timestamp` (5-minute intervals)
- Split Series: Terms on `fingerprint.keyword` to see operation types
→ [View example in Playground](https://play.oodle.ai/logs/app/visualize#/edit/900d7c27-e8f8-4d95-9b47-a1773733a924)
---
### Area Chart
Area charts show magnitude over time with filled regions. Stacked area charts
display how parts contribute to a whole.
**Best for:**
- Visualizing volume trends over time
- Showing proportional contributions from different sources
- Highlighting sudden changes in data volume
**Configuration:**
1. **Metric**: Count
2. **X-Axis**: Date Histogram on `timestamp`
3. **Split Series**: Terms bucket on a categorical field
4. In **Metrics & axes**, set Chart type to "Area" and enable "Stacked"
**Example: Log Volume by Container**
- Metric: Count
- X-Axis: Date Histogram on `timestamp` (5-minute intervals)
- Split Series: Terms on `container_name.keyword`
→ [View example in Playground](https://play.oodle.ai/logs/app/visualize#/edit/4dbd9c50-edb9-11f0-b051-df37cf89c1fe)
---
### Pie Chart
Pie and donut charts show proportional distribution of data across categories.
**Best for:**
- Showing relative proportions
- Distribution across a small number of categories (≤10)
- Understanding composition at a glance
**Configuration:**
1. **Metric**: Count (or Sum for numeric values)
2. **Split Slices**: Terms bucket on the field to segment by
3. Optionally enable "Donut" mode in Options
**Example: Transactions by Customer Loyalty Level**
- Metric: Count
- Split Slices: Filters bucket with queries for each level:
- `message:Platinum` → label "Platinum"
- `message:Gold` → label "Gold"
- `message:Silver` → label "Silver"
- `message:Bronze` → label "Bronze"
→ [View example in Playground](https://play.oodle.ai/logs/app/visualize#/edit/23a5fa40-edbc-11f0-b901-edc57b66473a)
---
### Metric
Metric visualizations display a single prominent number, ideal for KPIs and
summary statistics.
**Best for:**
- Displaying key performance indicators
- Showing totals or averages at a glance
- Dashboard summary panels
**Configuration:**
1. **Metric**: Count, Sum, Average, or other aggregation
2. **Split Group** (optional): Terms bucket to show multiple metrics side by
side
**Example: Error Count by Service**
- Metric: Count
- Query filter: `log_level:error OR level:ERROR`
- Split Group: Terms on `container_name.keyword` (top 3-5)
→ [View example in Playground](https://play.oodle.ai/logs/app/visualize#/edit/8c3e6f40-edb9-11f0-b051-df37cf89c1fe)
---
### Data Table
Data tables display aggregated data in tabular format, perfect for detailed
breakdowns and exports.
**Best for:**
- Detailed data exploration
- Exporting aggregated data
- Showing exact counts and values
**Configuration:**
1. **Metric**: Count (and additional metrics as needed)
2. **Split Rows**: Terms bucket on the field to list
3. Configure sort order (descending for "top N" lists)
**Example: Top Log Fingerprints**
- Metric: Count
- Split Rows: Terms on `fingerprint.keyword` with size 10
- Sort by Count descending
→ [View example in Playground](https://play.oodle.ai/logs/app/visualize#/edit/7b780660-2b10-11f0-8d5a-b38158070b23)
---
### Heat Map
Heat maps use color intensity to represent value magnitude across two
dimensions, perfect for spotting patterns across time and categories.
**Best for:**
- Identifying activity patterns across time and categories
- Spotting outliers in multi-dimensional data
- Understanding when and where activity occurs
**Configuration:**
1. **Metric**: Count (determines color intensity)
2. **X-Axis**: Date Histogram on `timestamp`
3. **Y-Axis**: Terms bucket on the categorical field
**Example: Log Activity by Container Over Time**
- Metric: Count
- X-Axis: Date Histogram on `timestamp` (1-hour intervals)
- Y-Axis: Terms on `container_name.keyword`
→ [View example in Playground](https://play.oodle.ai/logs/app/visualize#/edit/0d169890-edba-11f0-b051-df37cf89c1fe)
---
## Filtering Data
You can filter visualization data in several ways:
### Query Bar
Enter Lucene or KQL queries to filter documents:
```
log_level:error AND container_name:frontend
```
### Filters Aggregation
Use Filters as a bucket type to create labeled segments based on queries:
```
# Filter 1
Query: message:success
Label: Successful
# Filter 2
Query: message:failed OR message:error
Label: Failed
```
### Time Filter
Use the time picker in the top navigation to adjust the time range for all
visualizations.
## Saving Visualizations
1. Click **Save** in the top menu
2. Enter a descriptive title
3. Optionally add to an existing dashboard
4. Click **Save**
Saved visualizations can be:
- Added to multiple dashboards
- Shared with team members
- Used as a template for similar visualizations
## Next Steps
- Combine your visualizations into
[dashboards](/dashboards/logs/log-dashboards) for comprehensive monitoring
views
- Explore the [logs interface](/logs-visualization) for ad-hoc log analysis
---
## Create a Dashboard
This guide walks you through creating a new dashboard in Oodle.
## Create a New Dashboard
1. Click **Dashboards** in the left sidebar.
2. Click the **New** button near the top right of the page.
3. Choose **New dashboard** from the dropdown menu.
Now you should have a new blank dashboard.
## Add a New Panel
1. Click **Add visualization** on the empty dashboard screen to add a new panel.
2. Click **oodle** on the "Select data source" dialog.
## Configure the Panel
Panels come with extensive configuration options. Let's take a look at the
configuration options for [Cluster CPU Utilization panel from Oodle Playground](
https://play.oodle.ai/grafana/d/k8s_views_global/kubernetes-views-global?editPanel=72):
1. See how the panel will look like on the dashboard in the **visualization preview**.
2. Choose the visualization type from the **visualization dropdown**.
3. Write **PromQL query** to fetch the data for the panel,
4. Or use the **query builder** UI to build the query.
5. **Customize the panel** both in appearance and functionality using the provided options.
---
## Dashboard Variables
Dashboard variables allow you to create interactive, dynamic dashboards. They
can be used in queries and panel titles to create dashboards that can be easily
modified to show different data.
## Create a Variable
To add a variable to your dashboard:
1. Click the **Edit** button in the top-right corner of the dashboard.
2. Click the **Settings** button that just appeared.
3. The **Settings** page shows up.
1. Choose the **Variables** tab.
2. This page has link to the detailed documentation for variables if needed.
3. Click **Add variable** button.
In the next section we'll take a deeper look at some of the most useful and commonly used
variable types.
## Types of Variables
Oodle supports all the variable types [supported by Grafana](
https://grafana.com/docs/grafana/latest/dashboards/variables/).
Among those, two most useful and commonly used are **Query**,
and **Ad hoc filters**.
While the [Grafana docs](
https://grafana.com/docs/grafana/latest/dashboards/variables/
) cover the technical details, here we'll focus on how to configure them, and how
to use them in dashboards.
### Query Variable
Query variables are variables that show up as a dropdown in the dashboard, where
the values shown in the dropdown are determined by running a PromQL query.
Let's look at the `namespace` variable in the [Kubernetes / Views / Namespaces](
https://play.oodle.ai/grafana/d/k8s_views_ns/kubernetes-views-namespaces
) dashboard on Oodle Playground.
#### Appearance
This variable appears as a dropdown in the dashboard, where you can select the
kubernetes namespace.
#### Configuration
Here's how the [configuration page for `namespace`](
https://play.oodle.ai/grafana/d/k8s_views_ns/kubernetes-views-namespaces?editview=variables&editIndex=2
) looks like:
1. **Name** specifies the name that will be used in the dashboard to reference the variable.
2. **Query** is used to fetch the variable values, which will be displayed in the dropdown.
In this example, it means:
1. Query the metric `kube_pod_info`
2. Only consider the timeseries where `cluster` label equals the value of `cluster` variable.
Note how configuration of `namespace` variable makes use of another variable, `cluster`.
Also note how `$cluster` is used to reference the `cluster` variable.
3. From the resulting timeseries, take the value of `namespace` label.
4. Take the distinct set of values from the `namespace` label, and populate the dropdown.
3. **Multi-value** allows to select multiple values from the dropdown.
4. **Include All option** allows to include the special "All" value in the dropdown.
5. **Preview of values** shows all the values that will be present in the dropdown.
#### Usage
The [Kubernetes Resource Count panel](
https://play.oodle.ai/grafana/d/k8s_views_ns/kubernetes-views-namespaces?editPanel=32
) uses the `namespace` variable by adding `$namespace` appropriately in all the queries
of the panel that need to filter by namespace:
### Ad-hoc Filters
#### Appearance
Ad-hoc filters show up as double-dropdowns in the dashboard, where you can select:
1. The PromQL label to filter by
2. The value of the label to filter by
#### Configuration
Ad-hoc filters are configured by choosing **Ad-hoc filters** in "Select variable type" dropdown.
#### Usage
Unlike query variables, which need to be referenced in the query using `$variable_name`,
ad-hoc filters don't need to be referenced in the query at all. Instead, they are
automatically applied to all the queries in the dashboard.
---
## Terraform for dashboards
Configure all your Grafana dashboards and folders through terraform.
## Advantages of managing dashboards as code
* They are version-controlled, making changes and rollbacks more transparent
and trackable.
* This approach allows you to standardize dashboard configurations across
environments, eliminating manual errors and drift between staging and
production. You can apply the same dashboard configuration across all your
deployments / environments.
* Terraform's declarative nature also makes it easy to spin up or tear down
dashboards alongside your infrastructure, ensuring your monitoring
visualizations evolve with your stack.
* Integrating dashboard management into your CI/CD pipeline means faster,
more reliable deployments, as your dashboards will always align with the
infrastructure they monitor.
## How
:::info
Use oodle's [Terraform Provider](
https://registry.terraform.io/providers/oodle-ai/oodle/latest/docs/resources/grafana_dashboard)
:::
### Prerequisites
To setup Oodle Dashboards via Terraform files, you'll need the following
values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon ->
`API Keys` page in your Oodle UI to find out. (Oodle UI links:
)
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings`
icon -> `API Keys` in your Oodle UI to choose an appropriate key.
(Oodle UI links: )
### Example usage
```hcl
terraform {
required_providers {
oodle = {
source = "registry.terraform.io/oodle-ai/oodle"
}
}
}
# These can also be set as environment variables:
# export OODLE_DEPLOYMENT=https://us1.oodle.ai/
# export OODLE_INSTANCE="my-instance"
# export OODLE_API_KEY="my-api-key"
provider "oodle" {
deployment_url = "https://us1.oodle.ai/"
instance = "my-instance"
api_key = "my-api-key"
}
# Example of creating a Grafana folder and dashboard together
# The dashboard references the folder's UID
resource "oodle_grafana_folder" "my_folder" {
title = "Test Folder"
}
resource "oodle_grafana_dashboard" "test_folder" {
folder = oodle_grafana_folder.my_folder.uid
config_json = jsonencode({
"title" : "My Dashboard Title",
"uid" : "my-dashboard-uid",
"schemaVersion" : 39,
"timezone" : "browser",
"panels" : [
{
"id" : 1,
"type" : "stat",
"title" : "Request Rate",
"gridPos" : {
"h" : 8,
"w" : 12,
"x" : 0,
"y" : 0
},
"targets" : [
{
"expr" : "sum(rate(http_requests_total[5m]))",
"refId" : "A"
}
]
},
{
"id" : 2,
"type" : "timeseries",
"title" : "Error Rate",
"gridPos" : {
"h" : 8,
"w" : 12,
"x" : 12,
"y" : 0
},
"targets" : [
{
"expr" : "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))",
"refId" : "A"
}
]
}
]
})
}
# Example of a dashboard with a commit message for version history
resource "oodle_grafana_dashboard" "service_dashboard" {
folder = oodle_grafana_folder.my_folder.uid
message = "Initial dashboard creation"
overwrite = true
config_json = jsonencode({
"title" : "Service Health",
"uid" : "service-health-dashboard",
"schemaVersion" : 39,
"time" : {
"from" : "now-6h",
"to" : "now"
},
"panels" : [
{
"id" : 1,
"type" : "gauge",
"title" : "Uptime",
"gridPos" : {
"h" : 8,
"w" : 8,
"x" : 0,
"y" : 0
},
"targets" : [
{
"expr" : "avg(up{job=\"my-service\"})",
"refId" : "A"
}
]
}
]
})
}
# Example without a folder (created at root level)
resource "oodle_grafana_dashboard" "root_dashboard" {
config_json = jsonencode({
"title" : "Root Level Dashboard",
"uid" : "root-level-dashboard",
"schemaVersion" : 39,
"panels" : []
})
}
```
Apply terraform configuration
```bash
terraform init
terraform apply
```
---
## Metrics Dashboards
Oodle dashboards are a way to query, visualize, and organize metrics to gain
deeper insights into your data at a glance.
## Overview
Let's take a look at the [Kubernetes Overview dashboard on Oodle Playground](
https://play.oodle.ai/grafana/d/k8s_views_global/kubernetes-views-global):
1. **Dashboards button** on the left sidebar provides quick access to Oodle
dashboards.
2. **Dashboard path** shows your current location in the dashboard hierarchy,
including folder structure and dashboard name. You can navigate back through
folders using this breadcrumb trail.
3. **Variables** allow you to filter and refine your view.
4. **Time controls** let you set the time range for dashboard data.
5. **Visualization panels** show the data using a variety of visualizations.
## Explore Further
export const FilteredDocCardList = () => {
const category = useCurrentSidebarCategory();
const filteredItems = category.items.filter(
item => item.label !== 'Metrics Dashboards'
);
return ;
};
---
## Database Hosts
The Hosts tab provides a host-level view of your monitored
database instances, showing key performance and resource
utilization metrics.
## Hosts Table
| Column | Description |
|--------|-------------|
| **Database Host** | Instance hostname or endpoint |
| **Queries/s** | Average queries per second |
| **Total Duration** | Cumulative query execution time |
| **Avg Duration** | Mean query execution time |
| **Max Duration** | Longest single query execution |
| **Rows/s** | Rows processed per second |
| **Avg CPU Utilization** | Average CPU usage percentage |
| **Max CPU Utilization** | Peak CPU usage percentage |
## Filters
The left-hand panel provides:
- **Database Type** — Filter by engine (PostgreSQL, MySQL).
Each type is shown with its logo icon.
- **Version** — Filter by database version.
- **Database Identifier** — Search by DB instance identifier.
## Click to Drill Down
Click a host row to open a detail drawer with:
- **Overview metrics** — CPU utilization, active connections,
queries per second, and replication lag over time.
- **Databases table** — Lists each database on the host with
its size, active connections, commits, rollbacks, and
conflict count.
- **Replication metrics** — Write lag, flush lag, replay lag,
and replication slot delay (when applicable).
## Use Cases
- **Capacity planning** — Identify hosts approaching CPU or
connection limits.
- **Performance debugging** — Correlate host-level resource
spikes with query duration increases.
- **Fleet management** — Compare performance across database
instances to find outliers.
## Related Pages
- [Database Queries](/database-monitoring/queries) — Aggregated
query performance metrics.
- [Database Samples](/database-monitoring/samples) — View
individual query execution events.
---
## Database Monitoring
Oodle Database Monitoring gives you deep visibility into your
database performance. Track query execution times, identify slow
queries, monitor host resource utilization, and analyze query
samples — all from a single interface.
## Getting Started
Navigate to **Database Monitoring** in the sidebar. The default
view opens on the **Queries** tab, which shows an aggregated view
of all database queries.
## Navigation
Database Monitoring is organized into three tabs:
| Tab | Description |
|-----|-------------|
| **[Queries](/database-monitoring/queries)** | Aggregated query performance metrics |
| **[Samples](/database-monitoring/samples)** | Individual query execution samples |
| **[Hosts](/database-monitoring/hosts)** | Database host health and resource usage |
## Filters
Each tab provides a filter sidebar on the left. The **Queries**
and **Samples** tabs share these filters:
- **Host** — Filter by database host.
- **Database** — Filter by database name.
- **User** — Filter by the database user that executed the query.
- **Tables** — Filter by referenced table names.
- **Commands** — Filter by SQL command type
(e.g., SELECT, INSERT).
The **Hosts** tab has its own filters:
- **Database Type** — Filter by engine (PostgreSQL, MySQL).
- **Version** — Filter by database version.
- **Database Identifier** — Search by DB instance identifier.
All tabs include a **time picker** to adjust the observation
window.
## Supported Databases
Oodle Database Monitoring supports:
- **PostgreSQL** (including Aurora PostgreSQL)
- **MySQL** (including Aurora MySQL)
## Best Practices
- **Start with the Queries tab** to identify the most expensive
queries by total duration.
- **Drill into Samples** when you need to see individual
execution plans or parameter values.
- **Monitor Hosts** to detect resource bottlenecks (CPU,
connections) that affect query performance.
- **Set alerts** on key database metrics (query duration, error
rate) to catch regressions early.
## Related Pages
- [Kubernetes](/kubernetes/) — Monitor Kubernetes clusters,
workloads, and pods.
- [Serverless](/serverless/) — Monitor AWS Lambda functions.
---
## Database Queries
The Queries tab shows aggregated performance metrics for every
normalized database query. Use it to identify slow queries, high-
volume queries, and queries that consume the most total database
time.
## Overview Chart
At the top of the page, a time-series chart visualizes the
selected metric over time. Use the dropdown to switch between:
- **Total Duration** — Cumulative execution time across all
executions.
- **Avg Duration** — Mean execution time per query call.
- **Count** — Number of executions.
## Queries Table
Below the chart, the table lists every normalized query:
| Column | Description |
|--------|-------------|
| **Query** | Normalized SQL statement (parameters replaced with placeholders) |
| **Count** | Number of times the query ran in the selected window |
| **Total Duration** | Cumulative execution time with a visual bar |
| **Avg Duration** | Average execution time |
| **Avg Rows** | Average number of rows returned or affected |
| **Cache Hit Ratio** | Percentage of executions served from cache |
| **% Duration** | Share of total database time consumed by this query |
### Sorting
Click any column header to sort. Sorting by **Total Duration**
(descending) is the most effective way to find the queries that
matter most.
### Search
Use the search bar to filter queries by SQL text.
### Column Selector
Click the column-selector button in the toolbar to toggle
additional columns on or off. Hidden by default are
block-level I/O columns such as Avg Shared Blocks Hit/Read,
Avg Local Blocks Hit/Read, and Avg Temp Blocks Read/Written.
These are useful for diagnosing I/O-heavy queries.
### Show Full Query
Toggle **Show Full Query** in the toolbar to expand truncated
SQL statements in the table so you can read the complete query
text without clicking into the detail drawer.
## Filters
The left-hand panel provides:
- **Host** — Filter by database host.
- **Database** — Filter by database name.
- **User** — Filter by the database user that executed
the query.
- **Tables** — Filter by referenced table names.
- **Commands** — Filter by SQL command type
(e.g., SELECT, INSERT).
## Click to Drill Down
Click any query row to see:
- A detailed execution timeline.
- Per-host breakdown of execution statistics.
- Link to related [Samples](/database-monitoring/samples)
for that query.
## Related Pages
- [Database Samples](/database-monitoring/samples) — View
individual query execution events.
- [Database Hosts](/database-monitoring/hosts) — Monitor
host-level performance.
---
## Database Samples
The Samples tab displays individual query execution events
captured from your databases. Unlike the aggregated Queries view,
Samples shows each execution with its exact timestamp, full SQL
statement, and the database instance it ran on.
## Samples Table
| Column | Description |
|--------|-------------|
| **Time** | Exact timestamp of the execution. |
| **Statement** | Normalized SQL statement (displayed in monospace font). |
| **Database Instance** | The host the query ran on. |
| **User** | The database user that executed the query. |
| **Wait Event Group** | The wait-event category — color-coded badges for CPU, IO, Network, and Lock. |
| **Database** | The database name. |
| **Duration** | Execution duration, color-coded by severity. |
| **Application** | The client application name (if reported). |
| **State** | Current state of the query (e.g., active, idle). |
:::tip
The table shows the latest 1,000 samples. If the limit is
reached, a note appears above the table. Narrow the time range
to load more samples.
:::
## Filters
The left-hand panel provides:
- **Host** — Filter by database host.
- **Database** — Filter by database name.
- **User** — Filter by the database user.
- **Tables** — Filter by referenced table names.
- **Commands** — Filter by SQL command type
(e.g., SELECT, INSERT).
### Search
Use the search bar to filter samples by SQL text, host,
database, or user.
## Click to Drill Down
Click any sample row to open a detail drawer showing the full
query text, execution plan (when available), and contextual
details like client application name and user.
## Use Cases
- **Debug slow queries** — Find specific executions that were
abnormally slow and inspect their details.
- **Audit activity** — Review which queries ran at a specific
time against a specific database.
- **Investigate deadlocks** — Examine the exact statements
involved in a deadlock event.
- **Validate deployments** — After a schema migration, verify
that new query patterns appear as expected.
## Related Pages
- [Database Queries](/database-monitoring/queries) — Aggregated
query performance metrics.
- [Database Hosts](/database-monitoring/hosts) — Monitor
host-level performance.
---
## Amazon ECS
Oodle provides a seamless integration with [Elastic Container Service](https://aws.amazon.com/ecs/) and [AWS Fargate](https://aws.amazon.com/fargate/) to ingest **metrics, logs, and traces** from your ECS cluster.
The quickest path for all three signals is the **Datadog Agent**, which ships
metrics, logs, and traces together; see [Datadog Agent](#datadog-agent) below. If
you only need logs, the [log routers](#configuration) (FireLens, OpenTelemetry,
CloudWatch) cover that on their own.
## Datadog Agent
If the [Datadog Agent](https://docs.datadoghq.com/integrations/ecs_fargate/) runs
in your ECS tasks (injected as a sidecar on **Fargate**, or as a daemon task on
**EC2**), point it at Oodle by adding environment variables to the **Agent
container**, with no application changes required. This carries **metrics, logs,
and traces** together, unlike the log-only routers below.
**Dual-write** (Datadog + Oodle) adds Oodle as additional endpoints; keep your
existing `DD_API_KEY` / `DD_SITE`:
```json
"environment": [
{
"name": "DD_ADDITIONAL_ENDPOINTS",
"value": "{\"\": [\"\"]}"
},
{ "name": "DD_USE_V3_API_SERIES_ENABLED", "value": "false" },
{
"name": "DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS",
"value": "[{\"api_key\": \"\", \"Host\": \"\", \"Port\": 443, \"is_reliable\": false}]"
},
{ "name": "DD_LOGS_CONFIG_FORCE_USE_HTTP", "value": "true" },
{
"name": "DD_APM_ADDITIONAL_ENDPOINTS",
"value": "{\"\": [\"\"]}"
}
]
```
**Single-write** (Oodle only) overrides the Agent's primary endpoints and
authenticates with your Oodle API key:
```json
"environment": [
{ "name": "DD_API_KEY", "value": "" },
{ "name": "DD_DD_URL", "value": "" },
{ "name": "DD_USE_V3_API_SERIES_ENABLED", "value": "false" },
{ "name": "DD_APM_DD_URL", "value": "" },
{ "name": "DD_LOGS_CONFIG_LOGS_DD_URL", "value": ":443" },
{ "name": "DD_LOGS_CONFIG_FORCE_USE_HTTP", "value": "true" }
]
```
:::note
On the **EC2** launch type, also set `DD_ORCHESTRATOR_EXPLORER_ENABLED=false`.
Orchestrator Explorer ships to Datadog's site-based process intake, which
`DD_DD_URL` cannot redirect to Oodle.
:::
For the full reference (Helm/Operator/YAML elsewhere, plus a runnable
`terraform-aws-ecs-datadog` example for Fargate and EC2), see the
[Datadog on ECS guide](/integrations/metrics/datadog#amazon-ecs).
## Configuration
To configure Oodle to ingest logs from your ECS cluster, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
- `OODLE_LOGS_HOST`: The Oodle Host for log ingestion. Go to `Settings` -> `Logs` -> `Fluent Bit` in your Oodle UI to find out.
- `OODLE_METRICS_HOST`: The Oodle Host for metrics ingestion. Go to `Settings` -> `Metrics` -> `Prometheus` in your Oodle UI to find out.
### Using AWS FireLens
If you are using [awslogs](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) log driver to send logs to CloudWatch, [AWS FireLens](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html) is recommended for routing
logs to external destinations. Oodle uses Fluent Bit with FireLens for routing logs.
#### Customer-Hosted Fluent Bit Configs
It is recommended to host Fluent Bit configs in the customer's S3 buckets as that allows maximum flexibility
in terms of being able to version-control the configs, and modifying the config for specific needs. It also simplifies
ECS task definitions as config files can be directly pulled from customer's S3 bucket.
1. Store your fluent bit config in a S3 bucket. Following is an example config which configures dual-write
of logs to CloudWatch and Oodle, and also send Fluent Bit internal metrics to Oodle. It also adds following filters:
- [multiline](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/filter-multiline-partial-message-mode)
to concatenate partial/split container logs due to container runtime split.
```conf
[INPUT]
name fluentbit_metrics
tag internal_metrics
scrape_interval 2
[FILTER]
name multiline
match *
multiline.key_content log
mode partial_message
[OUTPUT]
Name cloudwatch_logs
Match *
region ${AWS_REGION}
log_group_name ${CLOUDWATCH_LOG_GROUP}
log_stream_name ${CLOUDWATCH_LOG_STREAM}/${ECS_TASK_ID}
auto_create_group true
Retry_Limit 3
[OUTPUT]
Name http
Match *
Host ${OODLE_LOGS_HOST}
Port 443
URI /ingest/v1/logs
Header X-OODLE-INSTANCE ${OODLE_INSTANCE}
Header X-API-KEY ${OODLE_API_KEY}
Format json
Compress gzip
Json_date_key timestamp
Json_date_format iso8601
Tls On
Retry_Limit 3
[OUTPUT]
Name prometheus_remote_write
Match internal_metrics
Host ${OODLE_METRICS_HOST}
Port 443
Uri /v1/prometheus/${OODLE_INSTANCE}/write
Header X-API-KEY ${OODLE_API_KEY}
Log_response_payload True
Tls On
# These environment variables are added by aws-for-fluent-bit init container
add_label ecs_cluster ${ECS_CLUSTER}
add_label ecs_task_arn ${ECS_TASK_ARN}
add_label ecs_task_definition ${ECS_TASK_DEFINITION}
add_label ecs_task_id ${ECS_TASK_ID}
Retry_Limit 3
```
:::note
Depending on your needs, you can update the fluent bit config, e.g. to only send logs to Oodle, you can remove
`cloudwatch_logs` Output section in the config.
:::
2. In the ECS task definition, update the value of `logDriver` from `awslogs` to `awsfirelens` for the application
container.
```json
{
"logConfiguration": {
"logDriver": "awsfirelens"
}
}
```
3. In the ECS task definition, add a Fluent Bit sidecar container to handle log routing. Provide relevant
environment variables referenced in the fluent bit config, and provide fluent bit config via `aws_fluent_bit_init_s3_1`
environment variable.
```json
{
"name": "fluent-bit",
"image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.32.5.20250305",
"essential": false,
"memory": 200,
"environment": [
{
"name": "OODLE_INSTANCE",
"value": ""
},
{
"name": "OODLE_API_KEY",
"value": ""
},
{
"name": "OODLE_LOGS_HOST",
"value": ""
},
{
"name": "OODLE_METRICS_HOST",
"value": ""
},
{
"name": "CLOUDWATCH_LOG_GROUP",
"value": ""
},
{
"name": "CLOUDWATCH_LOG_STREAM",
"value": ""
},
{
"name": "aws_fluent_bit_init_s3_1",
"value": "/fluent-bit.conf>"
}
],
"firelensConfiguration": {
"type": "fluentbit"
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "",
"awslogs-group": "",
"awslogs-stream-prefix": ""
}
}
}
```
4. Ensure your ECS [task role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) has
permissions to download from S3 bucket.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetBucketLocation"
],
"Resource": "*"
}
]
}
```
:::note
If you want to restrict the permissions to the specific bucket where your Fluent Bit configs
are hosted, then you can update `Resource` to `arn:aws:s3:::/*`
:::
5. If you are writing logs from Fluent Bit to CloudWatch as well, then ensure your ECS task role has relevant
CloudWatch permissions as well.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:::log-group::*"
}
]
}
```
#### Oodle-Hosted Fluent Bit Configs
If you prefer to use Oodle-Hosted Fluent Bit configs, then you can follow steps in this section.
##### Send to CloudWatch and Oodle
In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs
are sent to CloudWatch.
1. In the application container definition, update the value of `logDriver` from `awslogs` to `awsfirelens`.
```json
{
"logConfiguration": {
"logDriver": "awsfirelens"
}
}
```
2. Add an ephemeral volume in the ECS task definition. This volume will be used to store Fluent Bit configuration file
as configured in Step #3.
```json
"volumes": [
{
"name": "config",
"host": {}
}
]
```
3. Add two containers to the ECS task definition:
- An init container to download Fluent Bit configuration file
- A Fluent Bit sidecar container to handle log routing to both CloudWatch and Oodle.
```json
{
"name": "config-init",
"image": "public.ecr.aws/docker/library/alpine:3.21.3",
"essential": false,
"memoryReservation": 64,
"command": [
"sh",
"-c",
"set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-cloudwatch-v3.conf || exit 1"
],
"mountPoints": [
{
"sourceVolume": "config",
"containerPath": "/oodle",
"readOnly": false
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "",
"awslogs-group": "",
"awslogs-stream-prefix": ""
}
}
},
{
"name": "fluent-bit",
"image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.32.5.20250305",
"essential": false,
"memory": 200,
"environment": [
{
"name": "OODLE_INSTANCE",
"value": ""
},
{
"name": "OODLE_API_KEY",
"value": ""
},
{
"name": "OODLE_LOGS_HOST",
"value": ""
},
{
"name": "OODLE_METRICS_HOST",
"value": ""
},
{
"name": "CLOUDWATCH_LOG_GROUP",
"value": ""
},
{
"name": "CLOUDWATCH_LOG_STREAM",
"value": ""
}
],
"mountPoints": [
{
"sourceVolume": "config",
"containerPath": "/oodle",
"readOnly": true
}
],
"dependsOn": [
{
"containerName": "config-init",
"condition": "COMPLETE"
}
],
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/oodle/fluent-bit.conf"
}
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "",
"awslogs-group": "",
"awslogs-stream-prefix": ""
}
}
}
```
4. Ensure your ECS [task role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)) has relevant
CloudWatch permissions.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:::log-group::*"
}
]
}
```
##### Send to Oodle Only
In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.
1. Configure the application container and volume (Steps 1-2 above).
2. Update the `config-init` container definition.
```json
{
"name": "config-init",
"image": "public.ecr.aws/docker/library/alpine:3.21.3",
"essential": false,
"memoryReservation": 64,
"command": [
"sh",
"-c",
"set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-v3.conf || exit 1"
],
"mountPoints": [
{
"sourceVolume": "config",
"containerPath": "/oodle",
"readOnly": false
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "",
"awslogs-group": "",
"awslogs-stream-prefix": ""
}
}
}
```
The `config-init` container downloads a simplified configuration file from [fluent-bit-v3.conf](https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-v3.conf)
that routes logs exclusively to Oodle.
In addition, environment variables `CLOUDWATCH_LOG_GROUP` and `CLOUDWATCH_LOG_STREAM` can be removed from
the `fluent-bit` container.
#### Optional Configurations
This section documents configurations which can be used if you have the specified use-cases.
##### JSON Logs
If your applications emit logs in json format, then you can add a fluent parser to parse the json logs
before routing them to destinations.
It makes use of [multi-config-support](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/multi-config-support)
and [parse-json](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/parse-json)
capability in AWS for Fluent Bit image.
To add `parse-json` parser in your `fluent-bit` configuration, you can add following environment variable to `fluent-bit`
container:
```json
{
"name": "aws_fluent_bit_init_file_1",
"value": "/fluent-bit/configs/parse-json.conf"
}
```
The default parser used in AWS for Fluent Bit image uses `Time_Format %d/%b/%Y:%H:%M:%S %z` to parse the timestamp from
the logs. If your application logs have a different timestamp format, you can override the json parser. As an example,
if your timestamp format is ISO8601 (`%Y-%m-%dT%H:%M:%S.%LZ`), then you can follow below steps:
1. Update `config-init` container command to download custom json parser related configuration files, add following `wget`
commands in the command used in `config-init` container:
```
wget -O /oodle/parser.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-parser-v1.conf && wget -O /oodle/parse-json.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-filter-parse-json-iso8601-v1.conf
```
2. Add following environment variables in `fluent-bit` container:
```
{
"name": "aws_fluent_bit_init_file_1",
"value": "/oodle/parser.conf"
},
{
"name": "aws_fluent_bit_init_file_2",
"value": "/oodle/parse-json.conf"
}
```
:::note
Parser definitions needs to be provided as separate file and cannot be merged
with Filter definitions in a single file.
You can choose to store these config files in your S3 bucket as well, and use
`aws_fluent_bit_init_s3_` environment variables to refer to them.
:::
##### Concatenate Partial/Split Container Logs
Container runtime splits log lines larger than 16KB into multiple log lines. Fluent Bit can concatenate these split
log lines using `multiline` filter. To use it, add following filter in your fluent bit configuration:
```conf
[FILTER]
name multiline
match *
multiline.key_content log
mode partial_message
```
:::note
This needs to be first filter in the pipeline so that all remaining filters operate on concatenated log lines.
:::
##### Enrich log lines with additional key-value pair
Say, you want to ingest your dev and prod cluster logs to Oodle. You can enrich each log line with a `environment: dev`
or `environment: prod` field by making following changes:
1. Update `config-init` container command to download custom modify filter configuration file, add following `wget` command
in the command used in `config-init` container:
```
wget -O /oodle/modify-add-field.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-modify-add-field-v1.conf
```
2. Add following environment variables in `fluent-bit` container:
```
{
"name": "FIELD_NAME",
"value": "environment"
},
{
"name": "FIELD_VALUE",
"value": "dev"
},
{
"name": "aws_fluent_bit_init_file_1",
"value": "/oodle/modify-add-field.conf"
},
```
:::note
The numbering in `aws_fluent_bit_init_file_*` can be used to add multiple init files, e.g. if you need both
json parsing and enrich field, then you can add all files needed with increasing counter.
:::
### Using OpenTelemetry Collector
Users can run [OpenTelemetry (OTel) Collector contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) image to send ECS logs to Oodle.
:::note
1. [AWS Distro for OpenTelemetry](https://aws-otel.github.io) image is not supported as it doesn't contain the [Fluent Forward Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/fluentforwardreceiver)
required by this configuration.
1. [Fluent Forward Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/fluentforwardreceiver)
is currenlty in **beta** status.
:::
#### Send to CloudWatch and Oodle
In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.
1. In the application container definition, update the value of `logDriver` from `awslogs` to `awsfirelens`.
```json
{
"logConfiguration": {
"logDriver": "awsfirelens"
}
}
```
Fluent Forward Receiver in OTel collector is used in this configuration. Therefore, the value of `logDriver`
is set to `awsfirelens` and the `firelensConfiguration` type in next step is set to `fluentbit`.
2. Add an OTel collector sidecar container in the ECS task definiton.
```json
{
"name": "otel-collector",
"image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
"essential": false,
"memory": 200,
"environment": [
{
"name": "OODLE_INSTANCE",
"value": ""
},
{
"name": "OODLE_API_KEY",
"value": ""
},
{
"name": "OODLE_LOGS_HOST",
"value": ""
},
{
"name": "CLOUDWATCH_LOG_GROUP",
"value": ""
},
{
"name": "CLOUDWATCH_LOG_STREAM",
"value": ""
}
],
"command": [
"--config",
"https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-cloudwatch-v2.yaml"
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "",
"awslogs-group": "",
"awslogs-stream-prefix": ""
}
},
"firelensConfiguration": {
"type": "fluentbit"
}
}
```
3. Ensure your ECS [task role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)) has relevant
CloudWatch permissions.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:::log-group::*"
}
]
}
```
OTeL collector downloads an [otel-config-cloudwatch-v2.yaml](https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-cloudwatch-v2.yaml)
from a public S3 repository owned by Oodle. The config files in this repository are versioned and immutable.
#### Send to Oodle Only
In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.
1. Configure the application container (Step 1 above).
2. Update the `otel-collector` container definition.
```json
{
"name": "otel-collector",
"image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
"essential": false,
"memory": 200,
"environment": [
{
"name": "OODLE_INSTANCE",
"value": ""
},
{
"name": "OODLE_API_KEY",
"value": ""
},
{
"name": "OODLE_LOGS_HOST",
"value": ""
}
],
"command": [
"--config",
"https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-v2.yaml"
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "",
"awslogs-group": "",
"awslogs-stream-prefix": ""
}
},
"firelensConfiguration": {
"type": "fluentbit"
}
}
```
The `otel-collector` container downloads a simplified configuration file from [otel-config-v2.yaml](https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-v2.yaml) that routes logs exclusively to Oodle.
In addition, environment variables `CLOUDWATCH_LOG_GROUP` and `CLOUDWATCH_LOG_STREAM` are also removed from the `otel-collector` container.
## CloudWatch Logs Integration
If you are already sending your ECS logs to CloudWatch and prefer to push logs from CloudWatch to Oodle instead of
modifying your ECS task definitions, you can use our CloudWatch integration.
This approach allows you to keep your existing logging setup while still getting your logs into Oodle.
For detailed instructions on how to set up the CloudWatch to Oodle integration, please refer to
our [CloudWatch integration guide](/integrations/logs/cloudwatch).
---
## Evaluator Dependencies
Evaluator dependencies let you chain evaluation rules
into a directed acyclic graph (DAG). A dependent rule
only runs on spans where **all** of its dependencies
scored > 0, saving LLM costs by skipping spans that
fail earlier, cheaper checks.
## How it works
1. The eval-worker builds a DAG from each rule's
`dependsOnRuleIds` field.
2. Rules are sorted into topological levels —
level 0 has no dependencies, level 1 depends only
on level 0, and so on.
3. After each rule runs, the eval-worker records which
span IDs had any score greater than zero.
4. Before running a dependent rule, the worker filters
candidates to only spans that passed **all** of
that rule's dependencies (AND logic).
5. Each rule still applies its own dedup, sampling,
and rate limits on the filtered set.
### What counts as "passing"
A span passes a dependency if **any** numeric score
returned by that rule is greater than zero. Categorical
scores are ignored.
## Example: code filter → LLM deep analysis
```
Code eval: has_output (level 0)
↓
LLM eval: Correctness (level 1)
↓
LLM eval: Deep analysis (level 2)
```
1. `has_output` is a code evaluator that returns 1 if
the span has non-empty output, 0 otherwise. It runs
on all matching spans.
2. `Correctness` depends on `has_output`. It only runs
the LLM judge on spans where the code eval scored
> 0 (i.e., spans that actually have output).
3. `Deep analysis` depends on `Correctness`. It only
runs on spans that passed the correctness check.
This saves LLM costs because the cheap code evaluator
filters out irrelevant spans before the expensive LLM
calls happen.
## Multi-dependency example
```
Code eval: has_output (level 0)
Code eval: is_english (level 0)
↓ ↓
LLM eval: Correctness (level 1)
```
`Correctness` depends on both `has_output` and
`is_english`. A span must pass **both** code evaluators
to be evaluated for correctness.
## Setting up dependencies
1. Go to **GenAI → Evaluators → New Evaluator** (or
edit an existing rule).
2. In the **Depends on** section, select one or more
existing rules.
3. The API validates that the new dependency does not
create a cycle. If it does, the save is rejected
with an error.
## Cycle prevention
The system blocks saving any rule that would create a
dependency cycle. For example, if rule A depends on
rule B, you cannot make rule B depend on rule A.
Cycle detection runs both server-side (at save time)
and is enforced by the API, which returns a 400 error
if a cycle is detected.
## Deleting a dependency target
If you try to delete a rule that other rules depend on,
the API returns a 409 Conflict error listing the
dependent rule names. You must first remove the
dependency from those rules before deleting the target.
## Visualizing dependencies
### Dependencies tab
The **Dependencies** tab on the Evaluators page shows
the full DAG across all rules. Each node represents a
rule with its name, type (Code or LLM), and status
(Active or Paused). Edges show the dependency direction.
Click any node to open the evaluator details drawer.
### Drawer subtree view
Inside the evaluator details drawer, the
**Dependencies** tab shows the subtree for the focused
rule — both its upstream dependencies and downstream
dependents. The focused node is highlighted.
## Best practices
- **Same filters recommended:** dependent rules should
share the same span filters for best results. The
eval-worker groups rules by filter set, so
cross-group dependencies work but span overlap may
be limited.
- **Use code evals as first-pass filters:** code
evaluators are fast and cheap. Use them to filter
out irrelevant spans before running LLM judges.
- **Keep the DAG shallow:** deeply nested dependencies
add latency to each eval cycle. Two to three levels
is usually sufficient.
---
## Evaluators
Evaluators automatically score your LLM spans to track
quality, accuracy, and compliance across your GenAI
application. Navigate to **Agent Observability →
Evaluators** ().
## Types of Evaluators
| Type | Best For | Execution | Guide |
|------|----------|-----------|-------|
| **[LLM-as-Judge](./llm-as-judge)** | Nuanced quality assessment, relevance, tone | Calls an LLM to evaluate each span | [Setup guide →](./llm-as-judge) |
| **Code** | Deterministic checks: regex, JSON schema, exact match, keyword detection | Runs Python code in an isolated microVM — fast and free | Coming soon |
## How It Works
1. **Create a template** — define what to evaluate
(LLM prompt or Python code)
2. **Create a rule** — configure which spans to
evaluate, sampling rate, and filters
3. **Scores appear** on the Traces page, attached to
each evaluated span
## When to Use Each Type
**Use LLM-as-Judge when:**
- You need subjective assessment (relevance,
helpfulness, tone)
- The evaluation criteria are hard to express as code
- You want natural-language reasoning with each score
- You want to use one of the 8
[built-in templates](./llm-as-judge#built-in-templates)
(hallucination, relevance, correctness, etc.)
**Use Code Evaluators when:**
- You need deterministic, repeatable checks
- Speed and cost matter (no LLM API call needed)
- You're validating structure (JSON schema, required
fields, format)
- You want exact match, regex, or keyword detection
## Evaluators Page
The evaluators page has three tabs:
### Evaluators Tab
Shows all active evaluation rules. Each row displays
the rule name, status (Active / Paused), 24-hour
evaluation cost, and timestamps. Click a rule to open
a detail drawer with a score-over-time chart,
execution logs, and the full configuration.
Use the sidebar to filter by status (Active / Paused)
and evaluator name.
### Library Tab
Browse all available templates — both the 8 built-in
managed templates and any custom templates your team
has created. Click a template to see its full prompt
or code, variables, and the rules using it.
The built-in managed templates cover common evaluation
criteria:
- Hallucination
- Helpfulness
- Relevance
- Toxicity
- Correctness
- Conciseness
- Context Relevance (RAG)
- Faithfulness (RAG)
See the [LLM-as-Judge guide](./llm-as-judge) for
details on each template.
### Scores Tab
View all scores produced by evaluators. Filter by
time range, score name, value range, and span labels.
Click any row to open the corresponding trace detail.
## Prerequisites
- LLM-as-Judge evaluators require an
[LLM Connection](/agent-observability/llm-connections)
- Code evaluators have no external dependencies
- Creating evaluators requires the **Editor** or
**Admin** role
---
## LLM-as-Judge Evaluators
LLM-as-Judge evaluators use a language model to
score your GenAI spans. They are ideal for
subjective assessments like relevance, helpfulness,
and tone that are hard to express as deterministic
code.
## Built-in Templates
Oodle includes eight managed evaluator templates
ready to use out of the box. Each returns a score
from 0 to 1 with reasoning.
| Template | Variables | What it measures |
|----------|-----------|------------------|
| **Hallucination** | query, generation, context | Whether the output is grounded in the query and context |
| **Helpfulness** | query, generation | How helpful and complete the answer is |
| **Relevance** | query, generation | How relevant the output is to the query |
| **Toxicity** | generation | Presence of toxic content, hate speech, or profanity |
| **Correctness** | query, generation, ground_truth | Factual correctness against a known answer |
| **Conciseness** | query, generation | Brevity without losing completeness |
| **Context Relevance** | query, context | Whether retrieved context is relevant to the query (RAG) |
| **Faithfulness** | generation, context | Whether the output stays faithful to the context (RAG) |
Managed templates cannot be edited or deleted.
## Creating a Custom Template
1. Go to **GenAI → Evaluators**
()
2. Click **Create Evaluator** → **New Template**
3. Select **LLM as a Judge**
4. Enter a name and write your evaluation prompt
using `{{variable}}` placeholders
5. Click **Save**
The template automatically appends return
instructions that ask the model to respond with
a JSON score (0–1) and reasoning.
### Writing Effective Prompts
Use `{{variable}}` syntax to reference span data.
Common variables:
- `{{query}}` — the user's input
- `{{generation}}` — the model's output
- `{{context}}` — system instructions or
retrieved context
Example:
```text
You are an expert evaluator.
Score the following output on a scale of 0 to 1
based on whether it directly answers the question.
User query: {{query}}
LLM output: {{generation}}
```
## Creating an Evaluation Rule
After selecting a template (built-in or custom),
configure how it runs:
1. **Name** — the score name that appears on traces
2. **LLM Connection** — which provider to use for
evaluation
([configure here](/agent-observability/llm-connections))
3. **Model** — which model runs the evaluation
4. **Variable mapping** — map template variables to
span fields
5. **Sampling rate** — percentage of matching spans
to evaluate (1–100%, default 100%)
6. **Max invocations/hour** — rate cap to control
cost (default 100, set to 0 for no limit)
7. **Filter rules** — optional span label filters
to restrict which spans are evaluated
Click **Create** to activate the rule.
## Variable Mapping
Each `{{variable}}` in the template prompt maps to a
span field. Oodle auto-assigns sensible defaults:
| Variable | Default field |
|----------|---------------|
| `query`, `input` | `gen_ai.input.messages` |
| `generation`, `output` | `gen_ai.output.messages` |
| `context`, `system` | `gen_ai.system_instructions` |
| `ground_truth` | `expected_output` |
You can override any mapping by selecting a
different span tag from the dropdown. Add an
optional **JsonPath** expression (e.g.
`$[0].content`) to extract a nested value from
JSON fields.
Use the **Preview** toggle with a sample span to
verify that variables resolve correctly before
creating the rule.
## Managing Evaluators
### Evaluators Tab
The main tab shows all active evaluation rules.
Each row displays the rule name, status
(Active / Paused), 24-hour cost, and timestamps.
Click a rule to open a detail drawer with:
- **Score over time** chart (1h / 6h / 24h / 7d /
30d)
- **Execution logs** with per-span score, reasoning,
and trace links
- **Configuration** summary (connection, model,
template, mappings)
### Library Tab
Browse all templates — managed and custom. Click
any template to see its full prompt text, variables,
and the list of rules using it. Use **Use this
evaluator** to create a new rule from the template.
### Scores Tab
View all scores produced by evaluators across a
configurable time range. Filter by score name,
value range, and span labels. Click any row to open
the trace detail.
---
## GPU Monitoring
Oodle's GPU Monitoring gives you full visibility into
your GPU fleet — utilization, memory allocation,
temperature, power draw, and per-process resource
consumption. Works with NVIDIA GPUs using either the
`nvidia_gpu_exporter` (nvidia-smi based) or
`dcgm-exporter`.
## Overview Tab
The Overview tab provides a high-level health summary
of your entire GPU fleet.
### Summary Cards
| Card | Description |
|------|-------------|
| **Total GPUs** | Number of monitored GPU devices |
| **Active GPUs** | GPUs with utilization > 5% |
| **Total Hosts** | Number of hosts with GPUs |
| **Avg Utilization** | Fleet-wide average GPU utilization |
### Alerts & Health
| Alert | Trigger |
|-------|---------|
| **Idle GPUs** | Devices with < 5% utilization |
| **Thermal Throttling** | Devices with active thermal throttle or temp ≥ 85°C |
| **ECC / XID Errors** | Devices reporting memory or XID errors |
| **PCIe Degradation** | Active devices with zero PCIe throughput |
## Hosts Tab
The Hosts tab lists every host that has at least one
GPU installed.
| Column | Description |
|--------|-------------|
| **Host** | Hostname (instance label) |
| **Device** | GPU model name |
| **Health** | Healthy, Throttled, or Error state |
| **Active** | Number of active / total GPUs |
| **CPU** | Host CPU utilization % |
| **Memory** | Host system memory utilization % |
| **GPU** | Average GPU compute utilization % |
| **GPU Mem** | Average GPU VRAM allocation % |
| **ECC** | ECC error count |
| **XID** | XID error count |
Click any host row to open a detail drawer with
devices, processes, and embedded Grafana dashboards.
## Devices Tab
The Devices tab lists every individual GPU device across
all hosts.
| Column | Description |
|--------|-------------|
| **UUID** | GPU device UUID |
| **Model** | GPU model name |
| **Host** | Host the GPU is installed in |
| **Health** | Health status |
| **GPU Util** | Compute utilization % |
| **GPU Mem** | VRAM allocation % |
| **Temp** | Current temperature in °C |
| **Power** | Current power draw in watts |
| **ECC** | ECC error count |
Click any device row to open a detail drawer with
GPU-specific Grafana dashboards and process information.
## Process Monitor
The process monitor shows per-process resource usage on
GPU hosts using `process_exporter` metrics.
| Metric | Description |
|--------|-------------|
| **CPU Rate** | CPU cores consumed |
| **Resident Memory** | Physical memory usage |
| **Read / Write Bytes** | Disk I/O rate |
| **Context Switches** | Rate of context switches |
| **FD Ratio** | File descriptor usage as % of limit |
## Setup
Navigate to **Integrations → GPU Monitoring** to access
the setup wizard, which guides you through:
1. **Choose your exporter** — `nvidia_gpu_exporter`
or `dcgm-exporter`
2. **Install the exporter** — Docker, systemd, or
binary commands provided
3. **Install process exporter** — for per-process
visibility
4. **Configure scraping** — Prometheus or VMAgent
configuration with correct instance labeling
5. **Verify data** — confirm metrics are flowing
### Supported Exporters
| Exporter | Metrics Prefix | Use Case |
|----------|----------------|----------|
| **nvidia_gpu_exporter** | `nvidia_smi_*` | Simple setup, covers most use cases |
| **dcgm-exporter** | `DCGM_FI_*` | Advanced profiling metrics |
### Instance Label
The GPU page identifies hosts by the `instance` label.
The setup wizard configures relabeling to strip the port
from the target address, so you get clean hostnames
(e.g. `gpu-train-01` instead of `gpu-train-01:9835`).
## Grafana Dashboards
Four built-in dashboards are embedded in the UI,
accessible from host and device detail drawers:
| Dashboard | Used In |
|-----------|---------|
| **GPU Host Overview** | Host drawer |
| **NVIDIA GPU Metrics** | Device drawer (nvidia_smi) |
| **DCGM GPU Metrics** | Device drawer (DCGM) |
| **GPU Process Monitor** | Process tab |
The NVIDIA GPU Metrics dashboard provides a deep dive
into a single GPU — real-time utilization, clock speeds,
memory allocation, power draw, fan speed, and throttle
reasons. It automatically filters to the selected device
when opened from the Devices tab.
## Best Practices
- **Use consistent instance labels** — strip ports so
the same host appears as one entity across all metrics.
- **Monitor VRAM allocation** — high usage (>90%) leads
to OOM kills.
- **Watch thermal throttling** — sustained temps above
80°C reduce performance.
- **Track idle GPUs** — idle GPUs waste expensive
compute; use the alert to right-size your fleet.
- **Enable process_exporter** — without it, you lose
visibility into which processes consume resources.
---
## Oodle
Oodle is a fast, fully managed, drop-in replacement for Prometheus and ELK stack at scale.
Oodle helps engineering, and devops teams resolve incidents 5 times faster using AI-powered dashboard-less debugging. It delivers enterprise-grade observability at 1/5th the cost of enterprise-grade offerings using a game changing serverless architecture built from ground up on top of S3.
Oodle combines the security and reliability of enterprise-grade offerings with simplicity and cost-effectiveness of open source products with full open source compatibility with OpenTelemetry, Prometheus, Grafana, ELK stack. Achieve zero management with infinitely scalability.
Make your observability simple, fast and cost-efficient.
---
## Incident Bot
AI-powered incident response assistant that maintains
shared context across your team in Slack incident
channels. Every responder's question builds on prior
discoveries, so nobody wastes time re-explaining context
to the bot.
## Overview
The Incident Bot extends the
[Slack integration](/integrations/agents/slack) with
a fork-from-trunk session architecture designed for
incident channels:
- A **persistent trunk session** accumulates incident
knowledge as the investigation progresses
- Each **@mention forks** from the trunk for a parallel,
context-rich response
- **Discoveries are extracted** and propagated back to
the trunk so subsequent questions see prior findings
No additional Slack permissions are required beyond the
standard Slack integration.
## How It Works
### Automatic Detection
Incident channels are detected by name prefix. Any
channel whose name starts with one of these prefixes
is treated as an incident channel:
| Prefix | Example |
|--------|---------|
| `incident-` | `#incident-api-outage` |
| `inc-` | `#inc-db-failover` |
These prefixes are configurable. Contact
[Oodle support](mailto:support@oodle.ai) or use the
help chat widget in the sidebar to set custom prefixes
for your organization.
When the bot is mentioned in an incident channel, it
automatically switches to incident mode with shared
context.
### Fork-from-Trunk Architecture
```
Trunk Session (persistent)
├── Accumulates incident knowledge
├── Channel history & context
└── Extracted discoveries from all forks
Fork 1 (@mention by Alice)
├── Inherits full trunk context
├── Investigates Alice's question
└── Discoveries propagated back to trunk
Fork 2 (@mention by Bob)
├── Inherits trunk context + Alice's discoveries
├── Investigates Bob's question
└── Discoveries propagated back to trunk
```
Each fork runs in parallel, so multiple responders can
ask questions simultaneously without blocking each other.
## Prerequisites
- An active [Slack integration](/integrations/agents/slack)
with at least one connected workspace
- The `@Oodle` bot invited to the incident channel
## Getting Started
### Step 1: Install the Slack Integration
If you haven't already, install the Slack integration
by following the
[Slack setup guide](/integrations/agents/slack#installation).
### Step 2: Create an Incident Channel
Create a Slack channel with a recognized prefix
(defaults: `incident-`, `inc-`):
```
#incident-api-outage
#inc-database-failover
```
:::tip
Need different prefixes? Contact
[Oodle support](mailto:support@oodle.ai) or use the
help chat widget in the sidebar to configure custom
prefixes for your organization.
:::
### Step 3: Invite the Bot
Invite `@Oodle` to the incident channel:
```
/invite @Oodle
```
The bot detects the incident channel prefix and
activates incident mode automatically.
### Step 4: Start Investigating
Mention `@Oodle` with your questions. The bot will
use the full channel context and any prior discoveries
to give context-rich answers.
## Usage Examples
### Initial Investigation
```
@Oodle What errors are showing in the logs for the
payments service in the last 30 minutes?
```
### Building on Prior Context
After the first question surfaces relevant findings,
subsequent questions automatically include those
discoveries:
```
@Oodle Based on what we've found so far, what metrics
correlate with the error spike?
```
### Cross-Team Collaboration
Multiple responders can ask questions simultaneously.
Each response includes context from all prior findings:
```
Alice: @Oodle What changed in the last deploy?
Bob: @Oodle Are there any upstream dependency issues?
Carol: @Oodle Summarize what we know so far
```
### Incident Summary
```
@Oodle Give me a timeline of what happened in this
incident based on our investigation
```
## Best Practices
### Channel Naming
Use consistent prefixes across your organization so
the bot reliably detects incident channels. The default
prefixes are `incident-` and `inc-`, but these can be
customized — reach out to
[Oodle support](mailto:support@oodle.ai) to change them.
### Early Invitation
Invite `@Oodle` to the incident channel as early as
possible. The trunk session starts accumulating context
from the moment the bot joins, so earlier invitation
means richer context for all questions.
### Specificity
Be specific in your questions. Include service names,
time ranges, and what you're looking for:
```
@Oodle Show me error logs from checkout-service
between 14:00 and 14:30 UTC
```
### Building Context
Ask follow-up questions that build on prior findings.
The trunk accumulates discoveries, so questions like
"based on what we've found" leverage the full
investigation history.
## Troubleshooting
### Bot Not Using Incident Mode
Verify the channel name starts with a recognized
prefix (`incident-` or `inc-` by default). The
detection is based on the channel name prefix only.
If your organization uses custom prefixes, check with
your Oodle admin or
[contact support](mailto:support@oodle.ai).
### Missing Context from Earlier Questions
The trunk session persists for the lifetime of the
incident channel. If context seems missing, the bot
may not have been present when earlier messages were
sent. Invite `@Oodle` as early as possible.
---
## Slack Integration
Bring AI-powered observability insights directly into your Slack workspace.
Ask questions about your alerts, metrics, logs, and traces without leaving Slack.
## Overview
The Oodle AI Slack integration allows your team to:
- Query metrics, logs, and traces using natural language
- Get AI-powered insights and analysis
- Mention `@Oodle` in channels
- Use slash commands for quick queries
- [Incident Bot](/integrations/agents/incident-bot) —
shared-context AI responder for incident channels
## Features
### Natural Language Queries
Ask questions in plain English and get immediate insights:
- **Metrics**: "What is the error rate for my API?"
- **Logs**: "Show me logs from the last hour with errors"
- **Traces**: "What's causing high latency in the checkout service?"
- **Analysis**: "Why did CPU spike at 3pm today?"
### Agents
**Agent Mentions**
Mention `@Oodle` in any channel where the bot is added:
```
@Oodle What is the p99 latency for the auth service?
```
**Direct Messages**
Send queries directly to the Oodle AI bot:
```
Show me all ERROR logs from the payments service
```
**Slash Commands**
Use the `/oodle` slash command for quick queries:
```
/oodle analyze database performance
```
## Installation
### Prerequisites
- Admin access to your Slack workspace
- An active Oodle instance
### Step 1: Navigate to Integrations
You can find the Slack integration configuration by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Slack tile
### Step 2: Add Workspace
1. Click **Add Your First Workspace** (or **Add Workspace** if you already have workspaces connected)
1. You'll be redirected to Slack's OAuth page
2. Review the permissions requested
3. Click **Allow** to authorize the integration
### Step 3: Verify Installation
1. Return to the Oodle Integrations page
2. Your workspace should appear in the **Connected Workspaces** section
3. Open Slack and you should see Oodle AI in your Apps list
### Step 4: Add to Channels
1. In Slack, go to any channel where you want to use Oodle AI
2. Type `/invite @Oodle`
3. The bot will join the channel and be ready to answer questions
## Usage Examples
```
@Oodle What is the request rate for my API over the last hour?
```
```
@Oodle Show me ERROR logs from the payment service in the last 30 minutes
```
```
@Oodle What's causing high latency in the user-service?
```
```
/oodle summarize logs for last 15 minutes
```
## Best Practices
### Channel Strategy
**Public Channels**
Add Oodle AI to incident response or monitoring channels:
- `#incidents`
- `#alerts`
- `#on-call`
:::tip Incident Channels
For dedicated incident channels (prefixed with
`incident-` or `inc-`), the bot automatically
switches to incident mode with shared context across
all responders. See the
[Incident Bot](/integrations/agents/incident-bot)
documentation for details.
:::
### Team Collaboration
- Share queries and insights with your team in channels
- Use threads to keep discussions organized
- Pin important findings for future reference
### Query Tips
- Be specific with time ranges: "last hour", "past 24 hours", "today"
- Mention service names: "payment service", "auth API"
- Include severity levels: "ERROR logs", "critical alerts"
- Use natural language - no need for complex syntax
## Managing Workspaces
### View Connected Workspaces
1. Navigate to **Settings** > **Integrations** > **Slack**
2. See all connected workspaces with connection dates
### Add Multiple Workspaces
You can connect multiple Slack workspaces to the same Oodle instance:
1. Click **Add Workspace** in the Slack integration card
2. Authenticate with the new workspace
3. Both workspaces will appear in your connected list
### Remove a Workspace
To disconnect a Slack workspace:
1. Go to **Settings** > **Integrations** > **Slack**
2. Find the workspace you want to remove
3. Click **Remove**
4. Confirm the removal
:::warning
Removing a workspace will disconnect the bot immediately. You'll need to
reinstall to use Oodle AI again in that workspace.
:::
## Troubleshooting
### Bot Not Responding
**Check if bot is in the channel:**
```
/invite @Oodle
```
**Verify the workspace is still connected:**
1. Go to **Settings** > **Integrations** > **Slack**
2. Check if your workspace appears in the connected list
3. If not, reinstall the integration
**Try a direct message:**
Send a simple query directly to the bot to test connectivity.
### "Permission Denied" Errors
This means you don't have access to the requested data in Oodle:
- Check your user permissions in Oodle
- Ask your Oodle admin to grant appropriate access
- Try querying data you know you have access to
### Slow Responses
- Complex queries may take longer to process
- Large time ranges increase processing time
- Try narrowing your query scope
### Can't Install in Workspace
Ensure you have:
- Admin permissions in your Slack workspace
- Access to the Oodle instance
- A stable internet connection
If issues persist, contact your Slack workspace admin.
## FAQ
**Q: Can I use Oodle AI in multiple Slack workspaces?**
A: Yes, you can connect multiple Slack workspaces to a single
Oodle instance.
**Q: Can I customize the bot's responses?**
A: The bot automatically formats responses for optimal readability in Slack.
Custom formatting is not currently supported.
**Q: What happens if I remove the integration?**
A: The bot will immediately stop responding in your workspace.
You can reinstall at any time to restore functionality.
**Q: Does the bot work in private channels?**
A: Yes, simply invite the bot to any private channel where you want to use it.
---
## Custom MCP Servers
Add any MCP-compatible server to Oodle to extend the AI Assistant's capabilities
with additional tools, resources, and context. This allows you to connect
proprietary systems, internal tools, or third-party services that support the
Model Context Protocol.
## What are Custom MCP Servers?
Custom MCP servers let you connect any service that implements the
[Model Context Protocol](https://modelcontextprotocol.io/) to Oodle. When
connected, the AI Assistant gains access to the tools and resources provided by
that server.
Use cases include:
- **Internal tools**: Connect company-specific tools and APIs
- **Third-party services**: Integrate with external platforms
- **Custom automation**: Add specialized debugging or analysis tools
- **Data sources**: Give the AI access to additional context
## Prerequisites
- An MCP-compatible server with an HTTP endpoint
- The server URL and any required authentication headers
- An active Oodle instance
## Adding a Custom MCP Server
### Step 1: Open the Integration
1. Log in to your Oodle instance
2. Navigate to **Settings** → **Integrations**
3. Find the **External MCP Servers** integration card
4. Click to open the configuration drawer
### Step 2: Add a New Server
1. Click **Add MCP Server**
2. Fill in the server details:
- **Name**: A descriptive name for the server
- **URL**: The HTTP endpoint of the MCP server
- **Description** (optional): What the server provides
- **Headers** (optional): Authentication headers (e.g., API keys)
- **Additional Metadata** (optional): Context for the AI about when to use
this server
3. Click **Save**
### Step 3: Verify the Connection
After saving, Oodle automatically tests the connection and displays:
- **Tools**: Number of tools available from the server
- **Status**: Whether the server is reachable and responding
## Configuration Options
### Server URL
The HTTP endpoint where your MCP server is running. Must be accessible from
Oodle's servers.
```
https://your-mcp-server.example.com/mcp
```
### Authentication Headers
Add custom headers for authentication. Common patterns:
```json
{
"Authorization": "Bearer YOUR_API_KEY"
}
```
```json
{
"X-API-Key": "YOUR_API_KEY"
}
```
### Additional Metadata
Provide context to help the AI understand when to use this server:
```
Use this server for querying the customer database.
Available for production environment issues only.
```
## Managing MCP Servers
### View All Servers
The External MCP Servers integration shows a table of all configured servers
with:
- Server name and URL
- Number of available tools
- Connection status (Enabled/Disabled)
### Edit a Server
1. Click on a server row in the table
2. Modify the configuration
3. Click **Save**
### Test a Server
1. Click the three-dot menu on a server row
2. Select **Test**
3. View the connection result and available tools
### Delete a Server
1. Click the three-dot menu on a server row
2. Select **Delete**
3. Confirm the deletion
## Building Your Own MCP Server
To create a custom MCP server that works with Oodle:
1. Implement the [MCP specification](https://modelcontextprotocol.io/docs)
2. Expose an HTTP endpoint that handles MCP requests
3. Define the tools your server provides
4. Add authentication if needed
### Example Server Response
Your server should respond to tool listing requests with:
```json
{
"tools": [
{
"name": "search_database",
"description": "Search the customer database",
"inputSchema": {
"type": "object",
"properties": {
"query": { "type": "string" }
}
}
}
]
}
```
## Troubleshooting
### Server shows "Error" status
- Verify the URL is correct and accessible
- Check that authentication headers are properly formatted
- Ensure the server is running and responding to requests
- Check server logs for error details
### Tools not appearing
- Verify the server implements the MCP tools endpoint
- Check that the server returns valid JSON
- Ensure tools have proper names and descriptions
### Connection timeout
- Check network connectivity between Oodle and your server
- Verify firewalls allow the connection
- Ensure the server responds within the timeout period
### Authentication failures
- Verify API keys and tokens are correct
- Check header names match what the server expects
- Ensure tokens haven't expired
## Security Best Practices
- **Use HTTPS**: Always use encrypted connections
- **Rotate credentials**: Regularly update API keys and tokens
- **Limit scope**: Only grant the minimum required permissions
- **Monitor access**: Review AI interactions with external servers
- **Network security**: Use firewalls and IP allowlists where possible
---
## GitHub MCP Integration
Connect GitHub's Model Context Protocol (MCP) server to Oodle to enable the AI
Assistant to search and analyze your codebase. This integration allows Oodle to
**recommend code fixes when alerts are triggered**, helping you resolve issues
faster by understanding your code context.
## What is GitHub MCP?
GitHub provides an MCP server that exposes code search and repository access
capabilities. When connected to Oodle, the AI Assistant can:
- **Recommend code fixes** when alerts fire
- Search code across your repositories
- Read file contents for context
- Understand your codebase structure
- Provide code-aware debugging suggestions
## Prerequisites
- A GitHub account
- A GitHub Personal Access Token (PAT) with repository access
- An active Oodle instance
## Configuration
### Step 1: Generate a GitHub Personal Access Token
1. Go to [GitHub Personal Access Tokens](https://github.com/settings/personal-access-tokens)
2. Click **Generate new token**
3. Select **Fine-grained tokens** (recommended) or **Classic tokens**
4. Configure the token:
- **Name**: Give it a descriptive name (e.g., "Oodle MCP Integration")
- **Expiration**: Choose an appropriate expiration period
- **Repository access**: Select the repositories you want the AI to access
- **Permissions**: Grant `Contents` read access for code search
5. Click **Generate token**
6. Copy the token immediately (you won't be able to see it again)
:::warning
Keep your Personal Access Token secure. Never share it or commit it to version
control.
:::
### Step 2: Configure in Oodle
1. Log in to your Oodle instance
2. Navigate to **Settings** → **Integrations**
3. Find the **GitHub MCP** integration card under the MCP section
4. Click to open the configuration drawer
5. Enter your GitHub Personal Access Token
6. Wait for validation (a green checkmark indicates success)
7. Select the repositories you want the AI to access
8. Click **Save**
:::tip
The repository dropdown automatically loads repositories accessible with your
PAT. You can also manually type repository names in `owner/repo` format if
needed.
:::
## How It Works
Once configured, Oodle's AI Assistant can access your codebase when helping you
debug issues and **automatically suggest code fixes for alerts**. For example:
- **Alert-triggered code fixes**: When an alert fires, Oodle can analyze the
relevant code and suggest specific fixes to resolve the issue
- **Error investigation**: The AI can search your code for relevant functions,
error handlers, and configuration that may be causing issues
- **Context-aware suggestions**: By understanding your codebase structure, the
AI provides more accurate and actionable fix recommendations
- **Faster resolution**: Instead of just identifying problems, Oodle can show
you exactly what code to change
## Repository Selection
You can optionally specify which repositories the AI should focus on. This is
useful when:
- You want to limit AI access to specific projects
- You're debugging issues in a particular service
- You want faster responses by narrowing the search scope
If no repositories are selected, the AI can search across all repositories
accessible with your PAT.
## Security Considerations
- **Token Storage**: Your PAT is stored securely in Oodle's encrypted
configuration
- **Access Scope**: Only repositories you explicitly grant access to can be
searched
- **Token Rotation**: You can update or rotate your PAT at any time from the
integration settings
- **Audit Trail**: All AI interactions with your codebase are logged
### Recommended Token Permissions
For the GitHub MCP integration, we recommend:
| Permission | Access Level | Purpose |
|------------|--------------|---------|
| Contents | Read | Search and read code files |
| Metadata | Read | List repositories and branches |
Avoid granting write permissions unless specifically needed.
## Troubleshooting
### Token validation fails
- Verify the token hasn't expired
- Check that the token has the required permissions
- Ensure the token is correctly copied (no extra spaces)
### Repositories not loading
- Confirm the PAT has access to the repositories
- For organization repositories, ensure you've authorized the token for the org
- Try refreshing the page and re-entering the token
### AI can't find code
- Verify the correct repositories are selected
- Check that the files exist in the default branch
- Ensure the PAT has `Contents` read permission
## Managing the Integration
### Update Token
1. Open the GitHub MCP integration settings
2. Enter a new Personal Access Token
3. Click **Save**
### Change Repository Access
1. Open the GitHub MCP integration settings
2. Add or remove repositories from the selection
3. Click **Update**
### Disable Integration
1. Open the GitHub MCP integration settings
2. Toggle the **Enabled** switch off
3. Click **Update**
### Remove Integration
1. Open the GitHub MCP integration settings
2. Click the **Delete** button
3. Confirm the deletion
---
## External MCP Servers
Oodle can connect to external Model Context Protocol (MCP) servers to give the
AI Assistant additional context when helping you debug issues. By connecting
external tools and data sources, Oodle's AI can provide more accurate and
actionable recommendations.
## What are External MCP Servers?
While Oodle itself acts as an MCP server (allowing AI assistants like Cursor to
query your observability data), Oodle can also **connect to other MCP servers**
to enhance its own AI capabilities.
This allows Oodle's AI Assistant to:
- Access your codebase for context-aware debugging
- Recommend specific code fixes when alerts fire
- Understand your application architecture
- Provide more targeted troubleshooting guidance
## Available Integrations
- **[GitHub MCP](./github)** - Connect your repositories to enable AI-powered
code fix suggestions when alerts are triggered
- **[Custom MCP Servers](./custom-servers)** - Add any MCP-compatible server to
extend AI capabilities with custom tools
## How It Works
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ External MCP │ │ Oodle │ │ Observability │
│ Servers │────▶│ AI Assistant │◀────│ Data │
│ (GitHub, etc.) │ └─────────────────┘ └─────────────────┘
└─────────────────┘ │
▼
┌─────────────────────┐
│ Context-aware │
│ recommendations │
└─────────────────────┘
```
When investigating issues, Oodle's AI can:
1. Analyze the alert context (metrics, logs, traces)
2. Query connected external MCP servers for additional context
3. Search your codebase for relevant code (with GitHub MCP)
4. Access custom tools and data sources
5. Provide more accurate and actionable recommendations
## Getting Started
Connect an external MCP server to enhance Oodle's AI capabilities:
1. Navigate to **Settings** → **Integrations**
2. Find the external MCP server integration (e.g., GitHub MCP)
3. Follow the configuration steps for that integration
## Security
- External MCP connections are authenticated and encrypted
- You control which data sources the AI can access
- Credentials are stored securely in Oodle's configuration
- Access can be revoked at any time
---
## Integrations
### Easily integrate Oodle with your existing systems.
export const FilteredDocCardList = () => {
const category = useCurrentSidebarCategory();
const filteredItems = category.items.filter(item => item.label !== 'Integrations');
return ;
};
---
---
## Grafana Alloy
Oodle has [native support](./otel) for [Open Telemetry Logs](
https://opentelemetry.io/docs/concepts/signals/logs/).
To send logs to Oodle from Grafana Alloy, you will need the following:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
## Alloy Configuration
You can add a [`otelcol.exporter.otlphttp` component](
https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.exporter.otlphttp/)
to your Alloy configuration to send logs to Oodle:
```hcl
// git-new-start
otelcol.exporter.otlphttp "oodle_logs" {
client {
endpoint = "https://-logs.collector.oodle.ai"
headers = {
"X-OODLE-INSTANCE" = "",
"X-API-KEY" = "",
}
compression = "gzip"
}
logs_endpoint =
"https://-logs.collector.oodle.ai/ingest/otel/v1/logs"
}
// git-new-end
// Example of existing logs processor
otelcol.processor.attributes "add_environment" {
action {
// Existing config
}
output {
logs = [
otelcol.exporter.loki.default.input,
// git-new-start
otelcol.exporter.otlphttp.oodle_logs.input,
// git-new-end
]
}
}
```
### Steps
You can find the Alloy config changes by following these steps:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Alloy tile
3. Choose an appropriate API key from the list on top of the drawer
4. Select the Logs tab
Follow the steps in the drawer to add a [`otelcol.exporter.otlphttp` component](
https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.exporter.otlphttp/)
to your Alloy configuration to send logs to Oodle
---
## Auth0
Stream [Auth0](https://auth0.com/) authentication and security logs directly
to Oodle using Auth0's [Custom Log Streams](
https://auth0.com/docs/customize/log-streams/custom-log-streams).
Auth0 generates logs for authentication events, user management actions,
and security-related activities. By streaming these logs to Oodle, you can:
- Monitor authentication patterns and anomalies
- Track user sign-ups, logins, and password changes
- Detect security threats and suspicious activities
- Correlate auth events with your application logs
## Prerequisites
Before configuring Auth0 to send logs to Oodle, you'll need:
- An Auth0 account with access to Log Streams
- Your Oodle instance ID and API key (see [Configuration](#configuration))
## Configuration
To configure Auth0 log streaming, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Navigate to `Settings` icon →
`API Keys` page in your Oodle UI to find this value.
(Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Navigate to
`Settings` icon → `API Keys` in your Oodle UI to choose an appropriate key.
(Oodle UI links: )
### Steps
1. In Auth0, go to [Dashboard → Monitoring → Log Streams](
https://manage.auth0.com/#/log-streams) and click **Create Stream**.
2. Select **Custom Webhook**.
3. Configure the webhook with the following settings in **Log stream details**
section:
| Setting | Value |
|-------------------------|-------------------------------------------------------------------|
| **Name** | Enter a unique name for your new stream (e.g., "Oodle Logs"). |
| **Payload URL** | `https://-logs.collector.oodle.ai/ingest/v1/logs` |
| **Authorization Token** | `Bearer ` |
| **Content Type** | `application/json` |
4. [Optional] Configure the log event filters in the **Filter by Log Event
Category** section to filter which log events are sent to Oodle.
5. [Optional] Configure log obfuscation in the **Obscure log stream data**
section if you want to redact any of the following before it reaches Oodle:
- first name
- last name
- address
- email
- phone
- username
6. In **Payload preview** section, set **Content Format** to `JSON Array`.
7. Click **Save**.
8. Back on the [Log Streams page](https://manage.auth0.com/#/log-streams),
confirm the log stream status is **Active**.
---
## CloudWatch
Oodle provides a seamless integration with [AWS CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html).
It can be achieved by streaming CloudWatch Logs to [AWS Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html)
which delivers logs to Oodle.
The following diagram shows the data flow from CloudWatch Logs to Oodle.
## Configuration
### AWS Data Firehose Delivery Stream
To setup the integration, you need to create a AWS Data Firehose delivery stream
configured to deliver logs to Oodle endpoint. AWS Data Firehose put undelivered
logs to a backup S3 bucket. You will need following parameters to configure the integration:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
- `OODLE_ENDPOINT`: The Oodle endpoint URL for log ingestion. Note the endpoint URL must begin with "https://" per Firehose specification.
We provide a CloudFormation template to create the necessary resources in your AWS account.
You can use
[Quick Create](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://oodle-cf-templates.s3.us-west-2.amazonaws.com/log-stream-v1.yaml&stackName=oodle-log-stream-v1)
link to create the CloudFormation stack in your AWS account. You will be asked to enter
the parameters mentioned above. Be sure to choose the appropriate AWS region in which you want to create the stack.
Alternatively, you can download the template from [S3](https://oodle-cf-templates.s3.us-west-2.amazonaws.com/log-stream-v1.yaml)
and use it to create the resources in your AWS account.
The CloudFormation stack creates the following resources:
1. IAM Roles
- `LogStreamRole`: role to be assigned while creating subscription filter in log groups. Provides permission
to put logs into Firehose.
- `FirehoseRole`: role assigned to Firehose delivery stream, provides permission to write undelivered events
to S3 bucket.
2. S3 Bucket
- `S3BackupBucket`: bucket to hold undelivered logs from Firehose.
3. Firehose Delivery stream configured to send events to Oodle's http endpoint
### CloudWatch Logs Subscription Filter
After the CloudFormation stack is created, you need to create a CloudWatch Logs subscription
filter to deliver logs to the Data Firehose delivery stream. The subscription filter immediately starts
the flow of real-time log data from the chosen log group to your Amazon Data Firehose delivery stream:
```bash
aws logs put-subscription-filter \
--log-group-name "" \
--filter-name "Oodle" \
--filter-pattern "" \
--destination-arn "arn:aws:firehose:::deliverystream/" \
--role-arn "arn:aws:iam:::role/service-role/"
```
For the command above, you'll need to use the following parameters:
- ``: The name of your CloudWatch log group that contains the logs you want to send to Oodle
- ``: The AWS region in which the CloudFormation stack was created
- ``: Your AWS account ID
- ``: The name of the Firehose delivery stream created by the CloudFormation stack
- ``: The name of the LogStreamRole created by the CloudFormation stack
You can find the `` and `` in the Resources tab of the CloudFormation stack that was created:
1. Navigate to the [CloudFormation console](https://console.aws.amazon.com/cloudformation/home) in the AWS Management Console
2. Select the stack you created (e.g., "oodle-log-stream-v1")
3. Click on the "Resources" tab
4. Look for Physical ID with Logical ID:
- "LogStreamRole" to find the ``
- "Firehose" to find the ``
### Setup
1. Open `Settings` -> `Integrations` page in Oodle UI.
2. Choose `Amazon Web Services` integration tile.
3. Select the appropriate API key from the dropdown at the top of the drawer
4. Add your AWS account by entering AWS account ID and launching the Cloudformation Stack. The Cloudformation
stack creates an IAM role to provide Oodle access to your AWS account
5. Configure metrics to be collected in the Metrics tab. You can filter for specific AWS services
and also via tags to filter within a AWS service.
Oodle pre-configures dashboards for your selected services.
6. Configure CloudWatch logs to be collected in the Logs tab by following the instructions.
### Billing Considerations for Logs
You will incur [Amazon Data Firehose charges](https://aws.amazon.com/firehose/pricing/),
in addition to [your existing CloudWatch Logs costs](https://aws.amazon.com/cloudwatch/pricing/).
---
## Descope
Stream Descope audit events to Oodle using the Descope
[Open Telemetry Connector](
https://app.descope.com/connectors/template/opentelemetry
).
## Configuration
To configure the endpoint, you will need:
- `OODLE_INSTANCE`: Your Oodle instance ID.
- `OODLE_API_KEY`: Oodle API key with log ingestion permissions.
Find both at: `Settings` -> `API Keys`.
(Oodle UI links: ).
## Descope Open Telemetry Setup
### Configure a new Open Telemetry Connector
1. Click here to create a new Open Telemetry Connector:
[https://app.descope.com/connectors/template/opentelemetry](
https://app.descope.com/connectors/template/opentelemetry
)
2. Fill connector fields as shown below:
| Setting | Value |
|---|---|
| Logs Endpoint | `https://-logs.collector.oodle.ai/ingest/otel/v1/logs` |
| Protocol | `HTTP` |
| Authentication Type | `Bearer Token` |
| Token | `` |
| Request Header | `X-OODLE-INSTANCE: ` |
| Request Header | `X-OODLE-SOURCE: descope` |
| Stream Audit Events | `Enabled` |
| Stream Troubleshooting Events | `Enabled` (Optional) |
### Save and Verify
1. Click **Test** and confirm there are no errors.
2. Confirm the **Test Results** tab shows a success event.
3. Click **Save**.
4. Verify logs in Oodle Logs Explorer:
---
## Fluent Bit
Oodle provides a seamless integration with [Fluent Bit](https://fluentbit.io/). By adding Oodle as an output
destination in your Fluent Bit configuration, you can start sending logs directly to your Oodle instance.
## Configuration
To configure Fluent Bit to send logs to Oodle, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
- `OODLE_ENDPOINT`: The Oodle endpoint URL for log ingestion
Add the following configuration to your Fluent Bit configuration file:
```conf
# If a log line is getting split into multiple log lines, FluentBit can
# concatenate them with the "multiline" filter.
# Note that this needs to be first filter in the pipeline so that all remaining
# filters operate on concatenated log lines.
[FILTER]
name multiline
match *
multiline.key_content log
mode partial_message
[OUTPUT]
Name http
Match *
Host
Port 443
URI /ingest/v1/logs
Header X-OODLE-INSTANCE
Header X-API-KEY
Format json
Compress gzip
Json_date_key timestamp
Json_date_format iso8601
TLS On
```
### Setup
You can find the required Fluent Bit configuration changes by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Fluent Bit tile
3. Choose an appropriate API key from the list on top of the drawer
Add the configuration specified in the drawer to your Fluent Bit configuration file
## Collect Fluent Bit internal metrics (optional)
Monitor Fluent Bit itself (records processed, retries, output errors) by
collecting its internal metrics. The `fluentbit_metrics` input gathers them.
Push them straight to Oodle, or expose a `/metrics` endpoint for your own
Prometheus scraper to pull. Internal metrics flow through a separate pipeline
from logs, so your log filters do not touch them. See the
[Fluent Bit metrics docs](https://docs.fluentbit.io/manual/data-pipeline/inputs/fluentbit-metrics).
The `prometheus_remote_write` output pushes straight to Oodle.
```conf
[INPUT]
Name fluentbit_metrics
Tag internal_metrics
Scrape_Interval 30
[OUTPUT]
Name prometheus_remote_write
Match internal_metrics
Host .collector.oodle.ai
Port 443
Uri /v1/prometheus//write
Header X-API-KEY
Tls On
Tls.verify On
```
The `prometheus_exporter` output serves metrics at
`http://:2021/metrics`. Point your Prometheus compatible
scraper there and remote-write to Oodle.
```conf
[INPUT]
Name fluentbit_metrics
Tag internal_metrics
Scrape_Interval 30
[OUTPUT]
Name prometheus_exporter
Match internal_metrics
Host 0.0.0.0
Port 2021
```
---
## Fluentd
Oodle provides a seamless integration with [Fluentd](https://www.fluentd.org/). By adding Oodle as a output
destination in your Fluentd configuration, you can start sending logs directly to your Oodle instance.
## Configuration
To configure Fluentd to send logs to Oodle, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
Add the following configuration to your Fluentd configuration file:
```
@type http
endpoint "https://-logs.collector.oodle.ai/ingest/v1/logs"
headers {"X-OODLE-INSTANCE":"","X-API-KEY":""}
content_type application/json
json_array false
@type json
include_time_key true
time_key timestamp
time_format %Y-%m-%dT%H:%M:%S.%NZ
```
### Setup
You can find the required Fluent Bit configuration changes by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Fluentd tile
3. Choose an appropriate API key from the list on top of the drawer
Add the configuration specified in the drawer to your Fluentd configuration file
## Collect Fluentd internal metrics (optional)
Monitor Fluentd itself (buffer queue depth, retries, output errors) by exposing
its internal metrics. Install the `fluent-plugin-prometheus` gem
(`gem install fluent-plugin-prometheus`), then add these sources to expose a
Prometheus `/metrics` endpoint. See the
[Fluentd Prometheus metrics docs](https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus).
```
@type prometheus
bind 0.0.0.0
port 24231
metrics_path /metrics
@type prometheus_monitor
interval 30
@type prometheus_output_monitor
interval 30
```
Fluentd exposes these metrics for a Prometheus-compatible scraper to collect.
Point a scraper (Grafana Alloy, vmagent, or Prometheus in agent mode) at
`http://:24231/metrics` and remote-write to
`https://.collector.oodle.ai/v1/prometheus//write`
with the `X-API-KEY` header.
---
## HTTP
Send logs to Oodle using HTTP POST requests: from any application or system
that can make HTTP requests.
## Configuration
To send logs to Oodle via HTTP, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
### Steps
You can find the required HTTP configuration values by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the HTTP tile
3. Choose an appropriate API key from the list on top of the drawer
Execute the HTTP request via the possibilities listed in the tile (cURL, JavaScript, Python) to send logs to Oodle via HTTP POST requests.
## Sending Logs
Oodle can receive logs directly via HTTP POST requests. Here are a few minimal examples:
```bash
curl -X POST "https://-logs.collector.oodle.ai/ingest/v1/logs" \
-H "Content-Type: application/json" \
-H "X-OODLE-INSTANCE: " \
-H "X-API-KEY: " \
-d '{
"level": "info",
"timestamp": "2025-09-30T21:07:31.412Z",
"message": "Sample log message"
}'
```
```javascript
await fetch("https://-logs.collector.oodle.ai/ingest/v1/logs", {
method: "POST",
headers: {
"Content-Type": "application/json",
"X-OODLE-INSTANCE": "",
"X-API-KEY": ""
},
body: JSON.stringify({
level: "info",
timestamp: "2025-09-30T21:07:31.412Z",
message: "Sample log message"
})
})
```
```python
requests.post(
"https://-logs.collector.oodle.ai/ingest/v1/logs",
headers={
"Content-Type": "application/json",
"X-OODLE-INSTANCE": "",
"X-API-KEY": ""
},
json={
"level": "info",
"timestamp": "2025-09-30T21:07:31.412Z",
"message": "Sample log message"
}
)
```
## Log Format
Logs should be sent as JSON objects, with a `"message"` key.
```json
{
"message": "Your log message here",
"": ""
}
```
### Required Fields
- `message`: The log message content
### Supported Fields
The following table shows all fields Oodle supports. All of them are of type
`string`. Any field not listed here, if sent, will be captured in the `raw_log`
field, visible in Oodle Logs UI.
| Field Name | Description |
|------------|-------------|
| `message` | **Required.** The log message content |
| `timestamp` | Log timestamp |
| `level` | Log level (debug, info, warn, error, etc.) |
| `environment` | Environment name (prod, staging, dev, etc.) |
| `service` | Name of the service generating the log |
| `cluster` | Cluster identifier |
| `platform` | Platform identifier |
| `namespace` | Kubernetes namespace |
| `log_group` | Log group identifier |
| `log_stream` | Log stream identifier |
| `cloud_instance_id` | Cloud instance identifier |
| `node` | Node identifier |
| `container_id` | Container ID |
| `container_image` | Container image name |
| `container_name` | Container name |
| `namespace_labels` | Kubernetes namespace labels (JSON string) |
| `node_labels` | Node labels (JSON string) |
| `pod_labels` | Pod labels (JSON string) |
| `pod_annotations` | Pod annotations (JSON string) |
| `pod_ip` | Pod IP address |
| `pod_ips` | Pod IP addresses |
| `pod_name` | Pod name |
| `lambda_arn` | AWS Lambda ARN |
| `function_name` | Function name |
| `task_definition` | ECS task definition |
| `task_id` | Task identifier |
| `trace_id` | Distributed tracing trace ID |
| `span_id` | Distributed tracing span ID |
| `launch_type` | Launch type (e.g., EC2, FARGATE) |
| `source_type` | Source type identifier |
### Timestamp and Time Formats
If the request doesn't contain `timestamp`, the timestamp of when the request
was received is used.
The following formats are supported:
- RFC 3339: `2006-01-02T15:04:05Z07:00`
- ISO 8601: `20060102T15:04:05.999999Z07:00`
- Epoch seconds
- Epoch milliseconds
- Epoch microseconds
---
## Logs
### Easily integrate Oodle with your existing systems.
---
## Kubernetes
Instrument your kubernetes cluster in one command!
Collect metrics, logs, kubernetes events and eBPF powered APM + Service Graph data from your Kubernetes cluster.
## Installation
1. Open `Settings` -> `Integrations` page in Oodle UI.
2. Choose `Kubernetes` integration tile.
3. Select the appropriate API key from the dropdown at the top of the drawer
4. Enter your Kubernetes cluster name and pick all the telemetry signals you want to collect
5. Based on the selected signals, a helm values file is generated, copy the values file
and run the helm install commands to complete the integration
### Uninstallation
To uninstall the helm chart, run the following command.
```bash
helm uninstall -n oodle-monitoring oodle-observability
```
---
## Logstash
Oodle provides a seamless integration with [Logstash](https://www.elastic.co/logstash). By adding Oodle as an output
destination in your Logstash pipeline, you can start sending logs directly to your Oodle instance.
## Configuration
To configure Logstash to send logs to Oodle, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
Add the following output block to your Logstash pipeline configuration file:
```ruby
output {
http {
url => "https://-logs.collector.oodle.ai/ingest/v1/logs"
http_method => "post"
format => "json_batch"
content_type => "application/json"
headers => {
"X-OODLE-INSTANCE" => ""
"X-API-KEY" => ""
}
}
}
```
### Setup
You can find the required Logstash configuration changes by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Logstash tile
3. Choose an appropriate API key from the list on top of the drawer
Add the output configuration specified in the drawer to your Logstash pipeline configuration file
---
## Open Telemetry
Oodle provides a seamless integration with [Open Telemetry](https://opentelemetry.io/). By adding Oodle as an exporter
in your Open Telemetry configuration, you can start sending logs directly to your Oodle instance.
## Configuration
To configure Open Telemetry to send logs to Oodle, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
Add the following configuration to your Open Telemetry configuration file:
```yaml
exporters:
otlphttp/oodle:
endpoint: "https://"
headers:
"X-OODLE-INSTANCE": ""
"X-API-KEY": ""
```
### Steps
You can find the Open Telemetry config changes by following these steps:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Open Telemetry tile
3. Choose an appropriate API key from the list on top of the drawer
4. Ensure that you have selected the Logs feature in Step 1.
Follow the steps in the drawer to complete the OpenTelemetry setup
---
## Promtail
Oodle provides a seamless integration with [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/).
You can use [OTel Collector Loki receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/lokireceiver)
and then export the data via [OTel exporter](otel.mdx) to Oodle.
## Configuration
### OTel configuration
To configure Open Telemetry to send logs to Oodle, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
You can then configure OTel collector to receive logs from Promtail again via [Loki receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/lokireceiver)
```yaml
receivers:
loki:
protocols:
http:
endpoint: 0.0.0.0:3500
grpc:
endpoint: 0.0.0.0:3600
use_incoming_timestamp: true
exporters:
otlphttp/oodle:
logs_endpoint: "https://-logs.collector.oodle.ai/ingest/otel/v1/logs"
headers:
"X-OODLE-INSTANCE": ""
"X-API-KEY": ""
processors:
batch:
service:
pipelines:
logs:
receivers: [loki]
processors: [batch]
exporters: [otlphttp/oodle]
```
### Steps
You can find the required Promtail configuration changes by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Promtail tile
3. Choose an appropriate API key from the list on top of the drawer
You can then configure OTel collector to receive logs from Promtail again via [Loki receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/lokireceiver)
### Promtail configuration
You can now configure Promtail agent to send logs to OTel collector configured above:
```yaml
clients:
- url: http://otelcol:3500/loki/api/v1/push
```
---
## Query Logs
Query logs from Oodle using the OpenSearch-like HTTP API. This API supports
powerful filtering, sorting, and aggregations using OpenSearch Query DSL.
## Configuration
To query logs from Oodle via HTTP, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
- `INDEX_PATTERN`: Your logs index pattern Go to `Logs` icon -> `Explore` in your Oodle UI, you can find the index pattern in the top-left dropdown.
### Steps
You can find the required Query Logs configuration values by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Query Logs tile
3. Choose an appropriate API key from the list on top of the drawer
Follow the steps specified in the drawer to query Logs from Oodle via HTTP requests.
## Querying Logs
The API uses newline-delimited JSON (NDJSON) format with two lines:
1. Line 1: Index specification
2. Line 2: Search query using OpenSearch Query DSL
```bash
# Query logs using OpenSearch-like API
curl -X POST "https://.api.oodle.ai/api/v1/query_logs" \
-H "Content-Type: application/x-ndjson" \
-H "X-OODLE-INSTANCE: " \
-H "X-API-KEY: " \
-d '{"index": ""}
{"sort":[{"timestamp":{"order":"desc"}}],"size":100,"query":{"bool":{"filter":[{"range":{"timestamp":{"gte":1747913756000,"lte":1748000156000,"format":"epoch_millis"}}}]}}}'
```
```javascript
// Query logs using OpenSearch-like API
const queryPayload = { index: "" };
const searchPayload = {
sort: [{ timestamp: { order: "desc" } }],
size: 100,
query: {
bool: {
filter: [
{
range: {
timestamp: {
gte: 1747913756000,
lte: 1748000156000,
format: "epoch_millis"
}
}
}
]
}
}
};
const response = await fetch(
"https://.api.oodle.ai/api/v1/query_logs",
{
method: "POST",
headers: {
"Content-Type": "application/x-ndjson",
"X-OODLE-INSTANCE": "",
"X-API-KEY": ""
},
body: JSON.stringify(queryPayload) + "\n" + JSON.stringify(searchPayload)
}
);
const data = await response.json();
```
```python
# Query logs using OpenSearch-like API
query_payload = {"index": ""}
search_payload = {
"sort": [{"timestamp": {"order": "desc"}}],
"size": 100,
"query": {
"bool": {
"filter": [
{
"range": {
"timestamp": {
"gte": 1747913756000,
"lte": 1748000156000,
"format": "epoch_millis"
}
}
}
]
}
}
}
body = (
requests.compat.json.dumps(query_payload) + "\n" +
requests.compat.json.dumps(search_payload)
)
response = requests.post(
"https://.api.oodle.ai/api/v1/query_logs",
headers={
"Content-Type": "application/x-ndjson",
"X-OODLE-INSTANCE": "",
"X-API-KEY": ""
},
data=body
)
data = response.json()
```
## Query Format
The query uses NDJSON format where each JSON object must be on a single line:
### Line 1: Index Specification
```json
{"index": ""}
```
### Line 2: Search Query (OpenSearch-like DSL)
Use OpenSearch Query DSL to filter and sort logs. Common query components:
- **`query.bool.filter`** - Filter conditions
- **`query.bool.must`** - Required conditions
- **`query.bool.should`** - Optional conditions (OR logic)
- **`query.bool.must_not`** - Exclusion conditions
- **`sort`** - Sort order
- **`size`** - Number of results to return
- **`from`** - Pagination offset
### Common Query Examples
**Filter by log level:**
```json
{"query":{"bool":{"filter":[{"match_phrase":{"level.keyword":"ERROR"}}]}}}
```
**Filter by time range:**
```json
{"query":{"bool":{"filter":[{"range":{"timestamp":{"gte":1747913756000,"lte":1748000156000,"format":"epoch_millis"}}}]}}}
```
**Combine multiple filters:**
```json
{"query":{"bool":{"filter":[{"match_phrase":{"service.keyword":"api"}},{"range":{"timestamp":{"gte":1747913756000,"format":"epoch_millis"}}}]}}}
```
**Exclude specific values:**
```json
{"query":{"bool":{"filter":[{"match_phrase":{"level.keyword":"ERROR"}}],"must_not":[{"match_phrase":{"message":"health check"}}]}}}
```
## Response Format
The API returns a JSON response containing:
- `hits.hits[]` - Array of matching log entries
- `hits.total.value` - Total number of matching logs
- `took` - Query execution time in milliseconds
Example response:
```json
{
"took": 45,
"hits": {
"total": { "value": 234 },
"hits": [
{
"_source": {
"timestamp": "2025-01-20T10:30:00Z",
"level": "error",
"message": "Connection timeout",
"service": "api"
}
}
]
}
}
```
---
## Amazon S3
Oodle provides seamless integration with [Amazon S3](https://aws.amazon.com/s3/) for
ingesting logs stored in S3 buckets. This is useful for logs that are written directly
to S3, such as ALB access logs, CloudFront logs, or any application logs stored in S3.
The integration uses an SQS queue to receive S3 event notifications when new objects
are created. Oodle then reads the log files from S3 and ingests them.
## Configuration
### Prerequisites
Before setting up the integration, you'll need the following parameters from Oodle:
- `OodleAWSAccountId`: Oodle's AWS account ID
- `ExternalId`: A unique external ID provided by Oodle for secure cross-account access
Contact the Oodle team to obtain these values.
### Step 1: Deploy CloudFormation Stack
Deploy the CloudFormation stack that creates the necessary IAM role and SQS queue
for S3 log ingestion.
Use the [Quick Create](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://s3.us-west-2.amazonaws.com/oodle-configs/logs/s3/log-ingestion-v3.yaml&stackName=OodleS3Logs)
link to create the CloudFormation stack in your AWS account. Be sure to choose the
appropriate AWS region where your S3 buckets are located.
You will be asked to provide the following parameters:
| Parameter | Description |
|-----------|-------------|
| `OodleAWSAccountId` | Oodle's AWS account ID (provided by Oodle) |
| `ExternalId` | External ID for secure role assumption (provided by Oodle) |
| `S3BucketPattern1` | First S3 bucket name with optional prefix (required). Example: `my-logs-bucket` or `my-logs-bucket/alb-logs/` |
| `S3BucketPattern2-5` | Additional S3 bucket names with optional prefixes (optional) |
The CloudFormation stack creates the following resources:
1. **SQS Queue** (`OodleS3NotificationQueue-`): Receives S3 event notifications
when new log files are created
2. **IAM Role** (`OodleIntegrationS3LogIngestion-`): Allows Oodle to assume
this role to read logs from S3 and receive messages from the SQS queue
### Step 2: Share Stack Outputs with Oodle
After the CloudFormation stack is created:
1. Navigate to the [CloudFormation console](https://console.aws.amazon.com/cloudformation/home)
2. Select the stack you created (e.g., "OodleS3Logs")
3. Click on the **Outputs** tab
4. Share the following values with the Oodle team:
- `RoleARN`: ARN of the IAM role
- `QueueARN`: ARN of the SQS queue
### Step 3: Configure S3 Event Notifications
For each S3 bucket containing logs, you need to create an event notification to send
messages to the SQS queue when new objects are created.
1. Open the [S3 console](https://console.aws.amazon.com/s3/)
2. Select the bucket containing your logs
3. Navigate to the **Properties** tab
4. Scroll to the **Event notifications** section and click **Create event notification**
5. Configure the notification:
- **Event name**: Enter a descriptive name (e.g., "Oodle Logs")
- **Prefix** (optional): Add a path prefix if logs are in a specific directory
- **Event types**: Select **All object create events**
- **Destination**: Choose **SQS queue**
- **SQS queue**: Select **Choose from your SQS queues**
- **Queue**: Select `OodleS3NotificationQueue-`
6. Click **Save changes**
Repeat these steps for all buckets containing logs you want to ingest.
### Step 4: Verify the Integration
To verify that notifications are being sent to the SQS queue:
1. Open the [SQS console](https://console.aws.amazon.com/sqs/)
2. Search for `OodleS3NotificationQueue-`
3. Check that **Messages available** shows a value greater than 0
This confirms that S3 is successfully sending notifications when new log files are created.
Once Oodle team configures ingestion pipeline, logs would be visible in Oodle UI.
---
## Supabase
Stream Supabase project logs to Oodle using Supabase
[Supabase Log Drains](https://supabase.com/docs/guides/telemetry/log-drains).
## Configuration
To configure the endpoint, you will need:
- `OODLE_INSTANCE`: Your Oodle instance ID.
- `OODLE_API_KEY`: Oodle API key with log ingestion permissions.
Find both at: `Settings` -> `API Keys`.
(Oodle UI links: ).
## Supabase Log Drain Setup
### Create a Custom Endpoint
1. Open your Supabase project dashboard.
2. Go to **Project Settings** -> **Log Drains**.
3. Click **Custom Endpoint**.
### Configure the Custom Endpoint
Fill these destination fields:
| Setting | Value |
|---|---|
| URL | `https://-logs.collector.oodle.ai/ingest/v1/logs` |
| HTTP Version | `HTTP/1.1` or `HTTP/2` |
| Gzip | Enabled (recommended) |
Add these headers in the endpoint configuration:
| Header | Value |
|---|---|
| `X-OODLE-INSTANCE` | `` |
| `X-API-KEY` | `` |
### Save and Verify
1. Click **Save destination**.
2. Supabase will start sending logs to Oodle.
3. Verify logs in Oodle Logs Explorer:
---
## Vector
Oodle integrates seamlessly with [Vector](https://vector.dev/). By adding Oodle as a sink
in your Vector configuration, you can start sending logs directly to your Oodle instance.
## Configuration
To configure Vector to send logs to Oodle, you'll need the following values:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
Add the following configuration to your Vector configuration file:
```yaml
sinks:
oodle:
type: http
method: post
inputs: [ ]
uri: https://-logs.collector.oodle.ai/ingest/v1/logs
encoding:
codec: json
compression: gzip
request:
headers:
X-OODLE-INSTANCE: ""
X-API-KEY: ""
retry_attempts: 3
timeout_secs: 60
```
```toml
[sinks.oodle]
type = "http"
method = "post"
inputs = [ "" ]
uri = "https://-logs.collector.oodle.ai/ingest/v1/logs"
compression = "gzip"
encoding.codec = "json"
request.retry_attempts = 3
request.timeout_secs = 60
request.headers.X-OODLE-INSTANCE = ""
request.headers.X-API-KEY = ""
```
```json
{
"sinks": {
"oodle": {
"type": "http",
"method": "post",
"inputs": [ "" ],
"uri": "https://-logs.collector.oodle.ai/ingest/v1/logs",
"encoding": {
"codec": "json"
},
"compression": "gzip",
"request": {
"headers": {
"X-OODLE-INSTANCE": "",
"X-API-KEY": ""
},
"retry_attempts": 3,
"timeout_secs": 60
}
}
}
}
```
### Steps
You can find the Query Metrics commands and steps by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Vector tile
3. Choose an appropriate API key from the list on top of the drawer
Add the configuration specified in the drawer to your Vector configuration file
## Collect Vector internal metrics (optional)
Monitor Vector itself (throughput, buffer usage, component errors) by collecting
its internal metrics. The `internal_metrics` source gathers them. Push them
straight to Oodle, or expose a `/metrics` endpoint for your own Prometheus
scraper to pull. See the
[Vector `internal_metrics` docs](https://vector.dev/docs/reference/configuration/sources/internal_metrics/).
The `prometheus_remote_write` sink pushes straight to Oodle. It sends the API key
as a bearer token, which Oodle accepts.
```yaml
sources:
internal_metrics:
type: internal_metrics
scrape_interval_secs: 30
sinks:
oodle_metrics:
type: prometheus_remote_write
inputs: [ internal_metrics ]
endpoint: https://.collector.oodle.ai/v1/prometheus//write
compression: snappy
auth:
strategy: bearer
token: ""
```
```toml
[sources.internal_metrics]
type = "internal_metrics"
scrape_interval_secs = 30
[sinks.oodle_metrics]
type = "prometheus_remote_write"
inputs = [ "internal_metrics" ]
endpoint = "https://.collector.oodle.ai/v1/prometheus//write"
compression = "snappy"
auth.strategy = "bearer"
auth.token = ""
```
```json
{
"sources": {
"internal_metrics": {
"type": "internal_metrics",
"scrape_interval_secs": 30
}
},
"sinks": {
"oodle_metrics": {
"type": "prometheus_remote_write",
"inputs": [ "internal_metrics" ],
"endpoint": "https://.collector.oodle.ai/v1/prometheus//write",
"compression": "snappy",
"auth": {
"strategy": "bearer",
"token": ""
}
}
}
}
```
The `prometheus_exporter` sink serves metrics at
`http://:9598/metrics`. Point your Prometheus compatible scraper
there and remote-write to Oodle.
```yaml
sources:
internal_metrics:
type: internal_metrics
scrape_interval_secs: 30
sinks:
oodle_metrics_exporter:
type: prometheus_exporter
inputs: [ internal_metrics ]
address: 0.0.0.0:9598
```
```toml
[sources.internal_metrics]
type = "internal_metrics"
scrape_interval_secs = 30
[sinks.oodle_metrics_exporter]
type = "prometheus_exporter"
inputs = [ "internal_metrics" ]
address = "0.0.0.0:9598"
```
```json
{
"sources": {
"internal_metrics": {
"type": "internal_metrics",
"scrape_interval_secs": 30
}
},
"sinks": {
"oodle_metrics_exporter": {
"type": "prometheus_exporter",
"inputs": [ "internal_metrics" ],
"address": "0.0.0.0:9598"
}
}
}
```
---
## AvailableTools
The Oodle MCP server provides comprehensive observability tools across the
following categories:
### Logs
Query and filter logs with flexible time ranges and field-based filtering.
Discover available fields and run aggregations to analyze log patterns.
### Metrics
Execute PromQL queries to access your metrics data. Search for metrics,
explore labels, and validate queries.
### Traces
Search distributed traces by service, operation, or custom tags. Retrieve
specific trace details and explore available trace attributes.
### Alerts
Access active alerts, view monitor configurations, and check historical
trigger data to understand your alerting landscape.
### Grafana
Interact with Grafana dashboards programmatically.
Search, retrieve, update dashboards, and query panel
data directly. Dashboard write operations (save/update)
require an Editor or Admin API key.
### RUM (Real User Monitoring)
Query user sessions, browse session events, and investigate
frontend issues and errors reported by the RUM SDK.
### Integrations
Discover available integrations and retrieve setup
specifications for onboarding new data sources.
---
## BestPractices
1. **Be specific**: Provide clear context in your queries
(time ranges, service names, etc.)
2. **Start broad**: Begin with general queries and narrow down as needed
3. **Use natural language**: The AI assistant understands context, no need
for exact syntax
4. **Combine tools**: Ask questions that might require multiple tools
(e.g., "Show me logs and metrics for errors")
5. **Iterate**: If results aren't what you expected, refine your query with
more context
---
## CommonTroubleshooting
### Authentication errors
1. Verify your API key is correct
2. Check that your instance ID matches your Oodle instance
3. Ensure your API key has not expired
4. Try regenerating your API key from the Oodle UI
### Tool execution errors
1. Verify your Oodle instance is accessible
2. Check that your API key role has access to the
requested tool — Viewer keys cannot call write tools
(e.g., save/update Grafana dashboards)
3. Ensure your query parameters are valid
(e.g., valid time ranges)
### Connection refused
1. Check your internet connection
2. Verify the Oodle API endpoint URL is correct
3. Ensure no firewall or proxy is blocking the connection
---
## Toolsets
### Filtering Tools by Category
By default, all tools are available when you connect to the Oodle MCP server.
To limit the server to specific tool categories, append the `toolsets` query
parameter to the MCP URL:
```
https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp?toolsets=logs,metrics
```
Available toolsets:
| Toolset | Description |
|---------|-------------|
| `logs` | Log querying, filtering, field discovery, and aggregations |
| `metrics` | PromQL queries, metric/label discovery, query validation |
| `traces` | Trace search, retrieval, and label exploration |
| `alerts` | Active alerts, monitors, trigger history, anomalies |
| `grafana` | Dashboard search, retrieval, and management |
| `rum` | Real User Monitoring sessions, issues, and errors |
| `integrations` | Integration discovery and setup specifications |
Specify multiple toolsets as a comma-separated list. For example, to expose
only logs and metrics tools:
```
?toolsets=logs,metrics
```
:::tip
Limiting toolsets reduces the number of tools exposed to the AI assistant,
which can improve response quality by reducing context noise when you only
need a subset of capabilities.
:::
---
## UsageExamples
### Query Logs
```
Show me error logs from the last hour
```
```
What are the most common error messages in the auth-service?
```
```
Find all logs containing "timeout" from yesterday
```
### Query Metrics
```
What's the current CPU usage across all services?
```
```
Show me the request rate for the API service over the last 24 hours
```
```
Is there any metric that shows memory spikes?
```
### Query Traces
```
Show me slow traces from the checkout service
```
```
Find traces that took longer than 5 seconds
```
```
Get the trace with ID abc123xyz
```
### Check Alerts
```
Are there any active alerts right now?
```
```
Show me alerts that fired in the last 6 hours
```
```
What monitors are configured for the database service?
```
### Query Dashboards
```
Show me the dashboard for the API service
```
```
What's the current value of the "Request Rate" panel?
```
```
Find all dashboards related to Kubernetes
```
---
## Claude Code Integration
[Claude Code](https://www.anthropic.com/claude) is Anthropic's AI-powered
coding assistant. By integrating Oodle with Claude Code, you can query your
observability data directly from the AI assistant while coding.
## Prerequisites
- Claude Code installed on your machine
- An active Oodle instance
- Oodle API key (Admin, Editor, or Viewer)
## Configuration
### Step 1: Get Your API Key
1. Log in to your Oodle instance
2. Navigate to **Settings** → **Integrations**
3. Find the **Claude Code (MCP)** integration card
4. Copy the command from the configuration
### Step 2: Configure Claude Code
1. Run the following command in your terminal to add Oodle as an MCP server:
```bash
claude mcp add --transport http oodle-ai \
https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp \
--header "X-API-KEY: YOUR_API_KEY"
```
Replace:
- `YOUR_DEPLOYMENT` with your Oodle deployment domain
- `YOUR_INSTANCE` with your Oodle instance ID
- `YOUR_API_KEY` with your Oodle API key
:::tip
You can copy the exact command from the Claude Code (MCP) integration card in
your Oodle instance settings. It includes your instance ID and API key
pre-filled.
:::
### Step 3: Restart Claude Code
After adding the MCP server, restart Claude Code for the changes to take
effect.
### OAuth Configuration (Alternative)
Instead of using an API key, you can authenticate via OAuth. This uses the
standard MCP OAuth flow built into Claude Code:
```bash
claude mcp add --transport http \
--client-id YOUR_CLIENT_ID --callback-port 9400 \
oodle-ai https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp
```
After adding the server, run `/mcp` in Claude Code and follow the browser
login flow to complete authentication. The OAuth token is stored securely
in your system keychain.
## Configuration using Integration tile
You can find the required Claude Code configuration by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Claude Code tile
3. Choose an appropriate API key from the list on top of the drawer
Follow the steps specified in the drawer to integrate Oodle into your Claude Code
## Usage
Once configured, you can query your observability data directly in Claude
Code's AI assistant. Here are some example queries:
### Query Logs
```
Show me error logs from the last hour
```
```
What are the most common error messages in the auth-service?
```
```
Find all logs containing "timeout" from yesterday
```
### Query Metrics
```
What's the current CPU usage across all services?
```
```
Show me the request rate for the API service over the last 24 hours
```
```
Is there any metric that shows memory spikes?
```
### Query Traces
```
Show me slow traces from the checkout service
```
```
Find traces that took longer than 5 seconds
```
```
Get the trace with ID abc123xyz
```
### Check Alerts
```
Are there any active alerts right now?
```
```
Show me alerts that fired in the last 6 hours
```
```
What monitors are configured for the database service?
```
### Query Dashboards
```
Show me the dashboard for the API service
```
```
What's the current value of the "Request Rate" panel?
```
```
Find all dashboards related to Kubernetes
```
## Available Tools
The Oodle MCP server provides comprehensive observability tools across the
following categories:
### Logs
Query and filter logs with flexible time ranges and field-based filtering.
Discover available fields and run aggregations to analyze log patterns.
### Metrics
Execute PromQL queries to access your metrics data. Search for metrics,
explore labels, and validate queries.
### Traces
Search distributed traces by service, operation, or custom tags. Retrieve
specific trace details and explore available trace attributes.
### Alerts
Access active alerts, view monitor configurations, and check historical
trigger data to understand your alerting landscape.
### Grafana
Interact with Grafana dashboards programmatically.
Search, retrieve, update dashboards, and query panel
data directly. Dashboard write operations (save/update)
require an Editor or Admin API key.
### RUM (Real User Monitoring)
Query user sessions, browse session events, and investigate
frontend issues and errors reported by the RUM SDK.
### Integrations
Discover available integrations and retrieve setup
specifications for onboarding new data sources.
## Filtering Tools
### Filtering Tools by Category
By default, all tools are available when you connect to the Oodle MCP server.
To limit the server to specific tool categories, append the `toolsets` query
parameter to the MCP URL:
```
https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp?toolsets=logs,metrics
```
Available toolsets:
| Toolset | Description |
|---------|-------------|
| `logs` | Log querying, filtering, field discovery, and aggregations |
| `metrics` | PromQL queries, metric/label discovery, query validation |
| `traces` | Trace search, retrieval, and label exploration |
| `alerts` | Active alerts, monitors, trigger history, anomalies |
| `grafana` | Dashboard search, retrieval, and management |
| `rum` | Real User Monitoring sessions, issues, and errors |
| `integrations` | Integration discovery and setup specifications |
Specify multiple toolsets as a comma-separated list. For example, to expose
only logs and metrics tools:
```
?toolsets=logs,metrics
```
:::tip
Limiting toolsets reduces the number of tools exposed to the AI assistant,
which can improve response quality by reducing context noise when you only
need a subset of capabilities.
:::
## Troubleshooting
### Tools not showing up in Claude Code
1. Verify the MCP server was added successfully by running:
```bash
claude mcp list
```
2. Check that the API key is valid and has not expired
3. Ensure Claude Code was fully restarted after adding the MCP server
4. Check Claude Code's logs for any MCP-related errors
### Authentication errors
1. Verify your API key is correct
2. Check that your instance ID matches your Oodle instance
3. Ensure your API key has not expired
4. Try regenerating your API key from the Oodle UI
### Tool execution errors
1. Verify your Oodle instance is accessible
2. Check that your API key role has access to the
requested tool — Viewer keys cannot call write tools
(e.g., save/update Grafana dashboards)
3. Ensure your query parameters are valid
(e.g., valid time ranges)
### Connection refused
1. Check your internet connection
2. Verify the Oodle API endpoint URL is correct
3. Ensure no firewall or proxy is blocking the connection
## Best Practices
1. **Be specific**: Provide clear context in your queries
(time ranges, service names, etc.)
2. **Start broad**: Begin with general queries and narrow down as needed
3. **Use natural language**: The AI assistant understands context, no need
for exact syntax
4. **Combine tools**: Ask questions that might require multiple tools
(e.g., "Show me logs and metrics for errors")
5. **Iterate**: If results aren't what you expected, refine your query with
more context
## Security
- API keys are stored locally in Claude Code's configuration
- All communication is over HTTPS
- API keys can be rotated from the Oodle UI at any time
- Each API key is scoped to a specific Oodle instance
---
## Codex Integration
[Codex](https://openai.com/index/codex/) is OpenAI's AI-powered coding agent.
By integrating Oodle with Codex, you can query your observability data directly
from the AI assistant while coding.
## Prerequisites
- Codex CLI installed on your machine
- An active Oodle instance
## Configuration with OAuth (Recommended)
The Oodle CLI provides a one-command setup that handles OAuth authentication
automatically, including token refresh — no manual token management required.
### Step 1: Install the Oodle CLI
If you don't have the Oodle CLI installed, install it via Homebrew:
```bash
brew tap oodle-ai/oodle
brew install oodle
```
### Step 2: Set up Codex
```bash
oodle mcp setup codex --deployment DEPLOYMENT
```
Replace `DEPLOYMENT` with your deployment identifier (e.g., `us1`, `ap1`).
This opens a browser window for authentication (if needed) and patches
`~/.codex/config.toml` to run an Oodle MCP proxy via stdio. The proxy
injects a fresh OAuth token on every request and refreshes it automatically
when it expires.
### Step 3: Restart Codex
Restart Codex for the changes to take effect. That's it — no API keys or
tokens to copy.
:::tip
You can re-run `oodle mcp setup codex` at any time to update the
configuration. Use the `--name` flag to customise the MCP server name
(defaults to `oodle-ai`).
:::
## Configuration with API Key
If you prefer not to use the Oodle CLI, you can configure Codex with an
API key directly.
### Step 1: Get Your API Key
1. Log in to your Oodle instance
2. Navigate to **Settings** → **Integrations**
3. Find the **Codex (MCP)** integration card
4. Copy the configuration from the card
### Step 2: Configure Codex
Add the following to `~/.codex/config.toml`:
```toml
[mcp_servers.oodle-ai]
url = "https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp"
http_headers = { "X-API-KEY" = "YOUR_API_KEY" }
```
Replace:
- `YOUR_DEPLOYMENT` with your Oodle deployment domain
- `YOUR_INSTANCE` with your Oodle instance ID
- `YOUR_API_KEY` with your Oodle API key
:::tip
You can copy the exact configuration from the Codex (MCP) integration card in
your Oodle instance settings. It includes your instance ID and API key
pre-filled.
:::
### Step 3: Restart Codex
After saving `config.toml`, restart Codex for the changes to take effect.
## Usage
Once configured, you can query your observability data directly in Codex's AI
assistant. Here are some example queries:
### Query Logs
```
Show me error logs from the last hour
```
```
What are the most common error messages in the auth-service?
```
```
Find all logs containing "timeout" from yesterday
```
### Query Metrics
```
What's the current CPU usage across all services?
```
```
Show me the request rate for the API service over the last 24 hours
```
```
Is there any metric that shows memory spikes?
```
### Query Traces
```
Show me slow traces from the checkout service
```
```
Find traces that took longer than 5 seconds
```
```
Get the trace with ID abc123xyz
```
### Check Alerts
```
Are there any active alerts right now?
```
```
Show me alerts that fired in the last 6 hours
```
```
What monitors are configured for the database service?
```
### Query Dashboards
```
Show me the dashboard for the API service
```
```
What's the current value of the "Request Rate" panel?
```
```
Find all dashboards related to Kubernetes
```
## Available Tools
The Oodle MCP server provides comprehensive observability tools across the
following categories:
### Logs
Query and filter logs with flexible time ranges and field-based filtering.
Discover available fields and run aggregations to analyze log patterns.
### Metrics
Execute PromQL queries to access your metrics data. Search for metrics,
explore labels, and validate queries.
### Traces
Search distributed traces by service, operation, or custom tags. Retrieve
specific trace details and explore available trace attributes.
### Alerts
Access active alerts, view monitor configurations, and check historical
trigger data to understand your alerting landscape.
### Grafana
Interact with Grafana dashboards programmatically.
Search, retrieve, update dashboards, and query panel
data directly. Dashboard write operations (save/update)
require an Editor or Admin API key.
### RUM (Real User Monitoring)
Query user sessions, browse session events, and investigate
frontend issues and errors reported by the RUM SDK.
### Integrations
Discover available integrations and retrieve setup
specifications for onboarding new data sources.
## Filtering Tools
By default, all tools are available. To limit the MCP server to specific
tool categories, the method depends on how you configured Codex.
### With OAuth (Oodle CLI)
Pass the `--toolsets` flag when running setup:
```bash
oodle mcp setup codex --deployment DEPLOYMENT --toolsets logs,metrics
```
### With API Key
Append the `toolsets` query parameter to the MCP URL in `~/.codex/config.toml`:
```
https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp?toolsets=logs,metrics
```
### Available toolsets
| Toolset | Description |
|---------|-------------|
| `logs` | Log querying, filtering, field discovery, and aggregations |
| `metrics` | PromQL queries, metric/label discovery, query validation |
| `traces` | Trace search, retrieval, and label exploration |
| `alerts` | Active alerts, monitors, trigger history, anomalies |
| `grafana` | Dashboard search, retrieval, and management |
| `rum` | Real User Monitoring sessions, issues, and errors |
| `integrations` | Integration discovery and setup specifications |
:::tip
Limiting toolsets reduces the number of tools exposed to the AI assistant,
which can improve response quality by reducing context noise when you only
need a subset of capabilities.
:::
## Troubleshooting
### Tools not showing up in Codex
1. Verify the MCP server was added successfully by running:
```bash
codex mcp list
```
2. If using Oodle CLI setup, verify your auth is still valid:
```bash
oodle auth token
```
If expired, re-run `oodle mcp setup codex --deployment DEPLOYMENT`.
3. If using API key setup, check that the key is valid and has not expired
4. Ensure Codex was fully restarted after adding the MCP server
5. Check Codex's logs for any MCP-related errors
### Authentication errors
1. Verify your API key is correct
2. Check that your instance ID matches your Oodle instance
3. Ensure your API key has not expired
4. Try regenerating your API key from the Oodle UI
### Tool execution errors
1. Verify your Oodle instance is accessible
2. Check that your API key role has access to the
requested tool — Viewer keys cannot call write tools
(e.g., save/update Grafana dashboards)
3. Ensure your query parameters are valid
(e.g., valid time ranges)
### Connection refused
1. Check your internet connection
2. Verify the Oodle API endpoint URL is correct
3. Ensure no firewall or proxy is blocking the connection
## Best Practices
1. **Be specific**: Provide clear context in your queries
(time ranges, service names, etc.)
2. **Start broad**: Begin with general queries and narrow down as needed
3. **Use natural language**: The AI assistant understands context, no need
for exact syntax
4. **Combine tools**: Ask questions that might require multiple tools
(e.g., "Show me logs and metrics for errors")
5. **Iterate**: If results aren't what you expected, refine your query with
more context
## Security
- With the Oodle CLI setup, OAuth tokens are managed automatically and never
stored in Codex's configuration — the `oodle mcp serve` proxy injects them
at runtime
- With API key setup, keys are stored locally in `~/.codex/config.toml`
- All communication is over HTTPS
- API keys can be rotated from the Oodle UI at any time
- Each API key is scoped to a specific Oodle instance
---
## Cursor Integration
[Cursor](https://cursor.sh/) is an AI-first code editor. By integrating Oodle
with Cursor, you can query your observability data directly from the AI
assistant while coding.
## Prerequisites
- Cursor installed on your machine
- An active Oodle instance
- Oodle API key (Admin, Editor, or Viewer)
## Configuration
### Step 1: Get Your API Key
1. Log in to your Oodle instance
2. Navigate to **Settings** → **Integrations**
3. Find the **Model Context Protocol (MCP)** integration card
4. Copy your API key from the configuration
### Step 2: Configure Cursor
1. Open or create the MCP configuration file at `~/.cursor/mcp.json`
2. Add the Oodle MCP server configuration:
```json
{
"mcpServers": {
"oodle-ai": {
"type": "http",
"url": "https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp",
"headers": {
"X-API-KEY": "YOUR_API_KEY"
}
}
}
}
```
Replace:
- `YOUR_DEPLOYMENT` with your Oodle deployment domain
- `YOUR_INSTANCE` with your Oodle instance ID
- `YOUR_API_KEY` with your Oodle API key
:::tip
You can copy the exact configuration from the MCP integration card in your
Oodle instance settings. It includes your instance ID and API key pre-filled.
:::
### Step 3: Restart Cursor
After saving the configuration file, restart Cursor for the changes to take
effect.
## Configuration using Integration tile
You can find the required Cursor setup steps by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Cursor tile
3. Choose an appropriate API key from the list on top of the drawer
Follow the steps specified in the drawer to integrate Oodle into your Cursor IDE
## Usage
Once configured, you can query your observability data directly in Cursor's AI
assistant. Here are some example queries:
### Query Logs
```
Show me error logs from the last hour
```
```
What are the most common error messages in the auth-service?
```
```
Find all logs containing "timeout" from yesterday
```
### Query Metrics
```
What's the current CPU usage across all services?
```
```
Show me the request rate for the API service over the last 24 hours
```
```
Is there any metric that shows memory spikes?
```
### Query Traces
```
Show me slow traces from the checkout service
```
```
Find traces that took longer than 5 seconds
```
```
Get the trace with ID abc123xyz
```
### Check Alerts
```
Are there any active alerts right now?
```
```
Show me alerts that fired in the last 6 hours
```
```
What monitors are configured for the database service?
```
### Query Dashboards
```
Show me the dashboard for the API service
```
```
What's the current value of the "Request Rate" panel?
```
```
Find all dashboards related to Kubernetes
```
## Available Tools
The Oodle MCP server provides comprehensive observability tools across the
following categories:
### Logs
Query and filter logs with flexible time ranges and field-based filtering.
Discover available fields and run aggregations to analyze log patterns.
### Metrics
Execute PromQL queries to access your metrics data. Search for metrics,
explore labels, and validate queries.
### Traces
Search distributed traces by service, operation, or custom tags. Retrieve
specific trace details and explore available trace attributes.
### Alerts
Access active alerts, view monitor configurations, and check historical
trigger data to understand your alerting landscape.
### Grafana
Interact with Grafana dashboards programmatically.
Search, retrieve, update dashboards, and query panel
data directly. Dashboard write operations (save/update)
require an Editor or Admin API key.
### RUM (Real User Monitoring)
Query user sessions, browse session events, and investigate
frontend issues and errors reported by the RUM SDK.
### Integrations
Discover available integrations and retrieve setup
specifications for onboarding new data sources.
## Filtering Tools
### Filtering Tools by Category
By default, all tools are available when you connect to the Oodle MCP server.
To limit the server to specific tool categories, append the `toolsets` query
parameter to the MCP URL:
```
https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp?toolsets=logs,metrics
```
Available toolsets:
| Toolset | Description |
|---------|-------------|
| `logs` | Log querying, filtering, field discovery, and aggregations |
| `metrics` | PromQL queries, metric/label discovery, query validation |
| `traces` | Trace search, retrieval, and label exploration |
| `alerts` | Active alerts, monitors, trigger history, anomalies |
| `grafana` | Dashboard search, retrieval, and management |
| `rum` | Real User Monitoring sessions, issues, and errors |
| `integrations` | Integration discovery and setup specifications |
Specify multiple toolsets as a comma-separated list. For example, to expose
only logs and metrics tools:
```
?toolsets=logs,metrics
```
:::tip
Limiting toolsets reduces the number of tools exposed to the AI assistant,
which can improve response quality by reducing context noise when you only
need a subset of capabilities.
:::
## Troubleshooting
### Tools not showing up in Cursor
1. Verify the configuration file path is correct: `~/.cursor/mcp.json`
2. Check that the JSON is valid (no trailing commas, proper quotes)
3. Ensure Cursor was fully restarted after adding the configuration
4. Check Cursor's logs for any MCP-related errors
### Authentication errors
1. Verify your API key is correct
2. Check that your instance ID matches your Oodle instance
3. Ensure your API key has not expired
4. Try regenerating your API key from the Oodle UI
### Tool execution errors
1. Verify your Oodle instance is accessible
2. Check that your API key role has access to the
requested tool — Viewer keys cannot call write tools
(e.g., save/update Grafana dashboards)
3. Ensure your query parameters are valid
(e.g., valid time ranges)
### Connection refused
1. Check your internet connection
2. Verify the Oodle API endpoint URL is correct
3. Ensure no firewall or proxy is blocking the connection
## Best Practices
1. **Be specific**: Provide clear context in your queries
(time ranges, service names, etc.)
2. **Start broad**: Begin with general queries and narrow down as needed
3. **Use natural language**: The AI assistant understands context, no need
for exact syntax
4. **Combine tools**: Ask questions that might require multiple tools
(e.g., "Show me logs and metrics for errors")
5. **Iterate**: If results aren't what you expected, refine your query with
more context
## Security
- API keys are stored locally in your `~/.cursor/mcp.json` file
- All communication is over HTTPS
- API keys can be rotated from the Oodle UI at any time
- Each API key is scoped to a specific Oodle instance
---
## Generic MCP Server Integration
Use Oodle as a Model Context Protocol (MCP) server with any MCP-compatible AI
assistant. Query logs, metrics, traces, alerts, and Grafana dashboards using
natural language.
## What is MCP?
[Model Context Protocol](https://modelcontextprotocol.io/) is an open protocol
that standardizes how AI applications interact with external data sources. By
connecting Oodle as an MCP server, any compatible AI assistant can access your
observability data.
## Prerequisites
- An MCP-compatible AI assistant or client
- An active Oodle instance
- Oodle API key (Admin, Editor, or Viewer)
## Configuration
### Step 1: Get Your API Key
1. Log in to your Oodle instance
2. Navigate to **Settings** → **Integrations**
3. Find the **MCP Server** integration card
4. Copy your API key from the configuration
### Step 2: Configure Your MCP Client
The exact configuration steps depend on your AI assistant or MCP client. Most
clients use a configuration file (e.g., `mcp.json`, `config.json`).
Add the Oodle MCP server configuration:
```json
{
"mcpServers": {
"oodle-ai": {
"type": "http",
"url": "https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp",
"headers": {
"X-API-KEY": "YOUR_API_KEY"
}
}
}
}
```
Replace:
- `YOUR_DEPLOYMENT` with your Oodle deployment domain
- `YOUR_INSTANCE` with your Oodle instance ID
- `YOUR_API_KEY` with your Oodle API key
:::tip
You can copy the exact configuration from the MCP Server integration card in
your Oodle instance settings. It includes your instance ID and API key
pre-filled.
:::
### Step 3: Restart Your AI Assistant
After saving the configuration file, restart your AI assistant for the changes
to take effect.
## Configuration using Integration tile
You can find the required MCP Server configuration by doing the following:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the MCP Server tile
3. Choose an appropriate API key from the list on top of the drawer
Follow the steps speficied in the drawer to setup an MCP Server into your respective client.
The exact configuration steps depend on your AI assistant or MCP client.
Most clients use a configuration file (e.g., `mcp.json`, `config.json`).
## Usage
Once configured, you can query your observability data directly in your AI
assistant. Here are some example queries:
### Query Logs
```
Show me error logs from the last hour
```
```
What are the most common error messages in the auth-service?
```
```
Find all logs containing "timeout" from yesterday
```
### Query Metrics
```
What's the current CPU usage across all services?
```
```
Show me the request rate for the API service over the last 24 hours
```
```
Is there any metric that shows memory spikes?
```
### Query Traces
```
Show me slow traces from the checkout service
```
```
Find traces that took longer than 5 seconds
```
```
Get the trace with ID abc123xyz
```
### Check Alerts
```
Are there any active alerts right now?
```
```
Show me alerts that fired in the last 6 hours
```
```
What monitors are configured for the database service?
```
### Query Dashboards
```
Show me the dashboard for the API service
```
```
What's the current value of the "Request Rate" panel?
```
```
Find all dashboards related to Kubernetes
```
## Available Tools
The Oodle MCP server provides comprehensive observability tools across the
following categories:
### Logs
Query and filter logs with flexible time ranges and field-based filtering.
Discover available fields and run aggregations to analyze log patterns.
### Metrics
Execute PromQL queries to access your metrics data. Search for metrics,
explore labels, and validate queries.
### Traces
Search distributed traces by service, operation, or custom tags. Retrieve
specific trace details and explore available trace attributes.
### Alerts
Access active alerts, view monitor configurations, and check historical
trigger data to understand your alerting landscape.
### Grafana
Interact with Grafana dashboards programmatically.
Search, retrieve, update dashboards, and query panel
data directly. Dashboard write operations (save/update)
require an Editor or Admin API key.
### RUM (Real User Monitoring)
Query user sessions, browse session events, and investigate
frontend issues and errors reported by the RUM SDK.
### Integrations
Discover available integrations and retrieve setup
specifications for onboarding new data sources.
## Filtering Tools
### Filtering Tools by Category
By default, all tools are available when you connect to the Oodle MCP server.
To limit the server to specific tool categories, append the `toolsets` query
parameter to the MCP URL:
```
https://YOUR_DEPLOYMENT.oodle.ai/v1/api/instance/YOUR_INSTANCE/mcp?toolsets=logs,metrics
```
Available toolsets:
| Toolset | Description |
|---------|-------------|
| `logs` | Log querying, filtering, field discovery, and aggregations |
| `metrics` | PromQL queries, metric/label discovery, query validation |
| `traces` | Trace search, retrieval, and label exploration |
| `alerts` | Active alerts, monitors, trigger history, anomalies |
| `grafana` | Dashboard search, retrieval, and management |
| `rum` | Real User Monitoring sessions, issues, and errors |
| `integrations` | Integration discovery and setup specifications |
Specify multiple toolsets as a comma-separated list. For example, to expose
only logs and metrics tools:
```
?toolsets=logs,metrics
```
:::tip
Limiting toolsets reduces the number of tools exposed to the AI assistant,
which can improve response quality by reducing context noise when you only
need a subset of capabilities.
:::
## Supported AI Assistants
While Oodle works with any MCP-compatible client, here are some known
compatible assistants:
- **[Cursor](./cursor)** - AI-first code editor (see dedicated guide)
- **[Claude Code](./claude)** - Anthropic's AI-powered coding assistant (see dedicated guide)
- **[Codex](./codex)** - OpenAI's AI-powered coding agent (see dedicated guide)
- **Claude Desktop** - Anthropic's desktop application
- **Continue** - VS Code extension for AI assistance
- Any other tool that implements the MCP client specification
For specific setup instructions, consult your AI assistant's documentation on
adding MCP servers.
## Troubleshooting
### Tools not showing up
1. Verify the configuration file path and format
2. Check that the JSON is valid (no trailing commas, proper quotes)
3. Ensure your AI assistant was fully restarted after adding the configuration
4. Check your AI assistant's logs for any MCP-related errors
### Authentication errors
1. Verify your API key is correct
2. Check that your instance ID matches your Oodle instance
3. Ensure your API key has not expired
4. Try regenerating your API key from the Oodle UI
### Tool execution errors
1. Verify your Oodle instance is accessible
2. Check that your API key role has access to the
requested tool — Viewer keys cannot call write tools
(e.g., save/update Grafana dashboards)
3. Ensure your query parameters are valid
(e.g., valid time ranges)
### Connection refused
1. Check your internet connection
2. Verify the Oodle API endpoint URL is correct
3. Ensure no firewall or proxy is blocking the connection
## Best Practices
1. **Be specific**: Provide clear context in your queries
(time ranges, service names, etc.)
2. **Start broad**: Begin with general queries and narrow down as needed
3. **Use natural language**: The AI assistant understands context, no need
for exact syntax
4. **Combine tools**: Ask questions that might require multiple tools
(e.g., "Show me logs and metrics for errors")
5. **Iterate**: If results aren't what you expected, refine your query with
more context
## Security
- API keys are stored locally in your MCP client's configuration file
- All communication is over HTTPS
- API keys can be rotated from the Oodle UI at any time
- Each API key is scoped to a specific Oodle instance
---
## Model Context Protocol (MCP)
Oodle provides a Model Context Protocol (MCP) integration that enables AI
assistants like Cursor and Claude Code to directly access your observability
data. Query logs, metrics, traces, alerts, and Grafana dashboards using
natural language.
## What is MCP?
[Model Context Protocol](https://modelcontextprotocol.io/) is an open protocol
that standardizes how AI applications interact with external data sources. By
connecting Oodle through MCP, your AI assistant can:
- Query and filter logs
- Execute PromQL queries for metrics
- Search and analyze traces
- Check active alerts and monitor status
- Access and search Grafana dashboards
## Supported AI Assistants
Oodle's MCP integration works with:
- **[Cursor](./cursor)** - AI-first code editor
- **[Claude Code](./claude)** - Anthropic's AI-powered coding assistant
- **[Codex](./codex)** - OpenAI's AI-powered coding agent
- **[Any MCP-compatible client](./generic)** - Generic MCP server integration
As MCP adoption grows, more AI assistants will support the protocol.
## Available Tools
The Oodle MCP server provides comprehensive observability tools across the
following categories:
### Logs
Query and filter logs with flexible time ranges and field-based filtering.
Discover available fields and run aggregations to analyze log patterns.
### Metrics
Execute PromQL queries to access your metrics data. Search for metrics,
explore labels, and validate queries.
### Traces
Search distributed traces by service, operation, or custom tags. Retrieve
specific trace details and explore available trace attributes.
### Alerts
Access active alerts, view monitor configurations, and check historical
trigger data to understand your alerting landscape.
### Grafana
Interact with Grafana dashboards programmatically.
Search, retrieve, update dashboards, and query panel
data directly. Dashboard write operations (save/update)
require an Editor or Admin API key.
### RUM (Real User Monitoring)
Query user sessions, browse session events, and investigate
frontend issues and errors reported by the RUM SDK.
### Integrations
Discover available integrations and retrieve setup
specifications for onboarding new data sources.
## Authentication
MCP integration uses your Oodle API key for
authentication. The API key is configured once in your
AI assistant's MCP settings and used for all subsequent
requests.
API keys with any role — **Admin**, **Editor**, or
**Viewer** — can access the MCP integration. Viewer keys
are limited to read-only tools; Admin and Editor keys
have access to all tools including write operations
(e.g., saving or updating Grafana dashboards).
## Getting Started
Follow the integration guide for your AI assistant:
- [Cursor Integration](./cursor)
- [Claude Code Integration](./claude)
- [Codex Integration](./codex)
- [Generic MCP Server](./generic) - For any MCP-compatible AI assistant
:::tip
Want Oodle's AI to recommend code fixes? Connect external MCP servers like
[GitHub MCP](../external-mcp/github) to give the AI access to
your codebase.
:::
---
## Grafana Alloy(Alloy)
Oodle has native support for metrics emitted via the
[Prometheus Remote-Write Protocol](
https://prometheus.io/docs/specs/prw/remote_write_spec/).
To send metrics to Oodle from Grafana Alloy, you will need the following:
- `OODLE_INSTANCE`: Your Oodle instance ID. Go to `Settings` icon -> `API Keys` page in your Oodle UI to find out. (Oodle UI links: )
- `OODLE_API_KEY`: Your Oodle API key for authentication. Go to `Settings` icon -> `API Keys` in your Oodle UI to choose an appropriate key. (Oodle UI links: )
## Alloy Configuration
You can add a [`prometheus.remote_write` component](
https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.remote_write/)
to your Alloy configuration to send metrics to Oodle:
```hcl
// Example of existing prometheus destination
prometheus.remote_write "existing" {
endpoint {
// configuration ...
}
}
// git-new-start
prometheus.remote_write "oodle" {
endpoint {
url = "https://.collector.oodle.ai/v1/prometheus//write"
headers = {
"X-API-KEY" = "",
}
}
}
// git-new-end
// Example of existing prometheus forwarder
otelcol.exporter.prometheus "example" {
forward_to = [
prometheus.remote_write.existing.receiver,
// git-new-start
prometheus.remote_write.oodle.receiver,
// git-new-end
]
}
```
### Steps
You can find the Alloy Config config changes by following these steps:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Alloy tile
3. Choose an appropriate API key from the list on top of the drawer
4. Select the Metrics tab
Follow the steps in the Alloy tile drawer to send metrics to Oodle
---
## Service Graph with eBPF
Explore the new approach to unified debugging experience. Try out [our playground!](https://play.oodle.ai/service-graph)
Visualize service graphs, dependencies, APM, infrastructure metrics and logs with zero integration effort!
## Installation
1. Open settings page in Oodle UI.
2. Click on the `Kubernetes` integration tile and select `eBPF powered APM + Service Graph` option
3. Select the appropriate API key from the dropdown at the top of the drawer
4. Copy the values.yaml file provided in the integration tile, it configures Victoria Metrics agent
and Oodle's auto-instrumentation agent.
### Uninstallation
To uninstall the helm chart, run the following command.
```bash
helm uninstall -n oodle-monitoring oodle-observability
```
---
## Performance Requirements
The Oodle Auto Instrumenter uses Grafana's Beyla agent for eBPF-based instrumentation. This page outlines the performance requirements and resource considerations for running the agent in your environment.
## System Requirements
Before deploying the auto instrumenter, ensure your system meets these requirements:
- Linux kernel 5.8 or higher with BPF Type Format (BTF) enabled
- BTF is enabled by default on most Linux distributions with kernel 5.14 or higher
- Check if BTF is enabled by verifying if `/sys/kernel/btf/vmlinux` exists
- For RedHat-based distributions (RHEL8, CentOS 8, Rocky8, AlmaLinux8), kernel 4.18 with backported eBPF patches is supported
- eBPF must be enabled in the host
- For Go applications: Go 1.17 or higher (Beyla supports Go applications built with a major version no earlier than 3 versions behind the current stable release)
## Resource Requirements
### Recommended Requirements
- CPU: 200m (0.2 CPU cores)
- Memory: 2Gi
- Disk: 200Mi
## Performance Impact
The Beyla agent has been designed to minimize performance impact on your applications:
- **CPU Overhead**: Typically less than 1% of host CPU usage
- **Memory Usage**: Approximately 20-30MB base memory + 5-10MB per monitored process
- **Network Impact**: Minimal impact on network performance due to efficient eBPF-based collection
## Scaling Considerations
When deploying the auto instrumenter in production environments:
1. **Node Density**: The agent can monitor multiple processes on a single node. Recommended maximum is 50 processes per node.
2. **Network Traffic**: The agent can handle high network traffic volumes, but consider increasing resources if monitoring more than 10,000 requests per second.
3. **Metrics Volume**: The agent efficiently batches and compresses metrics before sending them to Oodle.
## Resource Configuration
You can adjust resource limits in your `values.yaml` when setting up beyla
1. Open settings page in Oodle UI.
2. Click on the `Kubernetes` integration tile and select `eBPF powered APM + Service Graph` option
3. Select the appropriate API key from the dropdown at the top of the drawer
4. Follow the steps in the drawer until you have generated a oodle-values.yaml file (in Step 7)
5. Make the following changes in the oodle-values.yaml file to specify resource limits:
```yaml
...
auto-instrumentation:
enabled: true
// git-new-start
beyla:
resources:
limits:
memory: 4Gi
requests:
cpu: 100m
memory: 1Gi
// git-new-end
...
```
6. Resume following the remaining steps in the drawer to complete the Kubernetes Service Graph setup with updated resource limits
## Limitations
Be aware of the following limitations:
1. **Instrumentation Scope**: Beyla provides generic metrics and transaction-level trace spans. For detailed code-level instrumentation, consider using language-specific agents.
2. **Permissions**: Beyla requires specific Linux capabilities:
- Basic operation: `CAP_DAC_READ_SEARCH`, `CAP_SYS_PTRACE`, `CAP_PERFMON`, `CAP_BPF`, `CAP_CHECKPOINT_RESTORE`, `CAP_NET_RAW`
- Network observability: Additional `CAP_NET_ADMIN` required
## Monitoring and Optimization
To ensure optimal performance:
1. Monitor the agent's resource usage using Kubernetes metrics
2. Adjust resource limits based on your specific workload
3. Consider using node selectors to deploy the agent on nodes with sufficient resources
4. Monitor internal metrics for performance insights
## Troubleshooting Performance Issues
If you experience performance issues:
1. Check the agent's resource usage and limits
2. Verify that the node has sufficient resources
3. Review the agent's logs for any warnings or errors
4. Consider reducing the number of monitored processes per node through beyla's selector configuration
5. Verify kernel version and BTF support
6. Check for any permission-related issues
## Additional Resources
For more detailed information about Beyla's performance characteristics and configuration options, refer to the [official Grafana Beyla documentation](https://grafana.com/docs/beyla/latest/).
---
## AWS CloudWatch
Oodle supports ingesting Amazon CloudWatch metrics and logs.
## Setup
1. Open `Settings` -> `Integrations` page in Oodle UI.
2. Choose `Amazon Web Services` integration tile.
3. Select the appropriate API key from the dropdown at the top of the drawer
4. Add your AWS account by entering AWS account ID and launching the Cloudformation Stack. The Cloudformation
stack creates an IAM role to provide Oodle access to your AWS account
5. Configure metrics to be collected in the Metrics tab. You can filter for specific AWS services
and also via tags to filter within a AWS service.
Oodle pre-configures dashboards for your selected services.
6. Configure CloudWatch logs to be collected in the Logs tab by following the instructions.
### Visualize CloudWatch metrics from API Integration
After setting up the data source, you can visualize your CloudWatch metrics in Oodle. Under your AWS account - open the `Metrics` tab, go to `Explore Metrics`
and choose your newly added CloudWatch data source. You should now be able to query and visualize your metrics.
## Metrics Streaming Integration
CloudWatch Metric Streams allows you to continually stream CloudWatch metrics to Oodle, with near-real-time delivery and low latency (3-5 minutes).
This streaming integration is powered by Amazon CloudWatch Metric Streams and Amazon Kinesis Data Firehose.
### Set up Metrics Streaming
1. We'll set up CloudWatch streaming and Amazon Kinesis Firehose using a CloudFormation template.
Download the template from [this link](https://oodle-cf-templates.s3.us-west-2.amazonaws.com/MetricStream.yaml)
2. Each AWS service has its own namespace and set of metrics. Here are some common AWS services and their namespaces:
- **AWS/EC2**: EC2 instance metrics like CPUUtilization, NetworkIn/Out, DiskReadOps/WriteOps
- **AWS/RDS**: Database metrics including DatabaseConnections, ReadIOPS, WriteIOPS, CPUUtilization
- **AWS/ECS**: Container metrics like CPUUtilization, MemoryUtilization, NetworkRxBytes/TxBytes
- **AWS/DynamoDB**: Database metrics including ConsumedReadCapacityUnits, ConsumedWriteCapacityUnits
- **AWS/ApplicationELB**: Load balancer metrics including RequestCount, TargetResponseTime, HTTPCode_Target_2XX_Count
You can identify the namespaces you want to monitor using one of these methods:
a. List all namespaces in your AWS account using the AWS CLI:
```bash
aws cloudwatch list-metrics --output json | jq -r '.Metrics[].Namespace' | sort -u
```
b. View specific namespaces by opening your [All Metrics](https://console.aws.amazon.com/cloudwatch/home#metricsV2) page in CloudWatch to see a list of all available namespaces for the services you are using
Choose the namespaces based on the AWS services you want to monitor. The corresponding metrics for those namespaces will be streamed to Oodle.
For more details on available metrics for each service, see [Viewing available metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html) in the AWS CloudWatch documentation.
3. Once you have identified the namespaces, modify the template downloaded in step 1 to specify which CloudWatch namespaces you want to monitor by adding them to the `MetricsStream.Properties.IncludeFilters` array.
Example:
```
MetricStream:
Type: AWS::CloudWatch::MetricStream
Properties:
.
.
.
IncludeFilters:
- Namespace: AWS/GlobalAccelerator
- Namespace: AWS/NetworkELB
- Namespace: AWS/ECS
- Namespace: AWS/DynamoDB
- Namespace: ECS/ContainerInsights
- Namespace: AWS/ElastiCache
- Namespace: AWS/ES
```
This example will stream metrics from Global Accelerator, Network Load Balancer, Amazon ECS, DynamoDB, ECS Container Insights, ElastiCache, and Elasticsearch Service to Oodle.
4. To run the template, you will need two parameters:
- OodleHttpEndpoint - The URL to send metrics to
- OodleApiKey - The API key for authentication
Navigate to the integrations page in your Oodle account and click the "View API Key" button in the top right to find the instance name and API key.
You can use the following URL for OodleHttpEndpoint: `https://{OODLE_INSTANCE}.collector.oodle.ai/v1/aws/${OODLE_INSTANCE}/metrics`
5. Deploy the CloudFormation template in your AWS account using the AWS Console, AWS CLI, or AWS SDK. Use the parameters from the previous step. For detailed instructions on creating a CloudFormation stack using the AWS Console, see [Creating a stack on the AWS CloudFormation console](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html).
**Note:** To add additional namespaces for monitoring later, you can modify the template directly through the AWS Console using Infrastructure Composer. For detailed instructions, see [Working with AWS CloudFormation Templates](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-get-template.html).
### Visualize CloudWatch Metrics from Metric Streaming
After setting up metric streaming, you can visualize your CloudWatch metrics in Oodle:
1. Go to `Explore Metrics`
2. Choose Oodle data source
3. Query and visualize your metrics
CloudWatch metrics are translated to Oodle metric names following this template:
`amazonaws_com_{namespace}_{metricName}_{aggregation}`
Note: Forward slashes (/) in the namespace are converted to underscores (_)
For example, the CloudWatch metric:
- Namespace: `AWS/ECS` (becomes `AWS_ECS`)
- Metric Name: `CPUUtilization`
Gets translated to these Oodle metrics:
- `amazonaws_com_AWS_ECS_CPUUtilization_sum` - Sum aggregation
- `amazonaws_com_AWS_ECS_CPUUtilization_count` - Count aggregation
- `amazonaws_com_AWS_ECS_CPUUtilization_quantile{quantile="0"}` - Minimum value
- `amazonaws_com_AWS_ECS_CPUUtilization_quantile{quantile="1"}` - Maximum value
### Billing Considerations for Metric Streaming
While Oodle does not charge extra for metric streaming, AWS pricing is based on metric updates and associated services. Each metric update includes four default statistics (Minimum, Maximum, Sample Count, and Sum). You will also incur Amazon Kinesis Data Firehose charges for data ingestion. For detailed pricing information, please refer to the "Metric streams" section under the Metrics tab on the [CloudWatch pricing page](https://aws.amazon.com/cloudwatch/pricing/).
### Alerting on CloudWatch Metrics
Oodle supports alerting on AWS CloudWatch metrics only when using the metric stream integration method.
To set up alerts on your CloudWatch metrics:
1. Ensure you have configured the metric stream integration as described above
2. Navigate to the Alerts section in Oodle
3. Create a new alert using your streamed CloudWatch metrics
The alert will use the translated metric names described in the previous section. For example, to alert on average ECS CPU utilization, you would use the query `amazonaws_com_AWS_ECS_CPUUtilization_sum/amazonaws_com_AWS_ECS_CPUUtilization_count `.
---
## Confluent Cloud
Oodle integrates with [Confluent Cloud](https://www.confluent.io/confluent-cloud/)
to pull metrics from the Confluent Cloud Metrics API. You
provide a Cloud API key and the resources you want to
watch, and Oodle collects Kafka cluster and connector
metrics and makes them available in your instance for
querying, dashboards, and alerting.
## Prerequisites
- A Confluent Cloud account
- A Confluent Cloud **Cloud API key** with the
**MetricsViewer** role, scoped to your organization or
environment (not to a single Kafka cluster)
- The IDs of the Kafka clusters (`lkc-...`) and, optionally,
connectors (`lcc-...`) you want to monitor
- An Oodle account (navigate to
to start setup)
## Setup
### 1. Create a Confluent Cloud API key
In the Confluent Cloud console, open
[API keys](https://confluent.cloud/settings/api-keys) and
create a **Cloud API key** scoped to your organization or
environment. Grant it the **MetricsViewer** role. A key
scoped to a single Kafka cluster does not work with the
Metrics API, so choose the organization or environment
scope.
### 2. Find your resource IDs
Collect the IDs of the resources you want to monitor:
| Resource | Where to find it | ID format |
|----------|------------------|-----------|
| Kafka cluster | Cluster Settings | `lkc-...` |
| Connector | Data Integration > Connectors | `lcc-...` |
### 3. Add the account in Oodle
1. Open the Confluent Cloud tile in Oodle
(),
or click **Settings** in the left sidebar and select
the **Confluent Cloud** tile.
2. Click **Add account** (the `+` button) and give the
account a name (for example `production`). The name
identifies this set of credentials in the list of
accounts.
3. Paste the **API Key** and **API Secret** you created.
4. Enter the **Cluster IDs** (`lkc-...`) and, optionally,
the **Connector IDs** (`lcc-...`) to monitor. Separate
multiple IDs with commas.
5. Click **Save**. Oodle validates the credentials against
the Confluent Metrics API and begins collecting metrics
within a few minutes.
## Verification
Once setup is complete, verify the integration is working:
1. In the Confluent Cloud tile, confirm the account status
shows as **Receiving**.
2. Go to and search for
metrics prefixed with `confluent_` to confirm data is
flowing.
3. In the tile, open **Visualize Data > View Dashboards**
and open the **Confluent Cloud Overview** dashboard.
## Metrics collected
Metrics are collected per configured cluster and connector.
Kafka metrics carry a `kafka_id` label, connector metrics
carry a `connector_id` label, and per-topic metrics carry a
`topic` label.
### Kafka cluster metrics
| Metric | Description |
|--------|-------------|
| `confluent_kafka_server_received_bytes` | Bytes produced to the cluster |
| `confluent_kafka_server_sent_bytes` | Bytes consumed from the cluster |
| `confluent_kafka_server_received_records` | Records produced to the cluster |
| `confluent_kafka_server_sent_records` | Records consumed from the cluster |
| `confluent_kafka_server_retained_bytes` | Bytes retained by the cluster |
| `confluent_kafka_server_active_connection_count` | Active client connections |
| `confluent_kafka_server_partition_count` | Partitions on the cluster |
| `confluent_kafka_server_request_count` | Requests by type |
| `confluent_kafka_server_consumer_lag_offsets` | Consumer group lag, by topic and group |
| `confluent_kafka_server_cluster_load_percent` | Estimated cluster load |
### Connector metrics
| Metric | Description |
|--------|-------------|
| `confluent_kafka_connect_sent_records` | Records sent by the connector |
| `confluent_kafka_connect_received_records` | Records received by the connector |
| `confluent_kafka_connect_sent_bytes` | Bytes sent by the connector |
| `confluent_kafka_connect_received_bytes` | Bytes received by the connector |
| `confluent_kafka_connect_dead_letter_queue_records` | Records routed to the dead letter queue |
## Dashboards
Oodle provides an out-of-the-box **Confluent Cloud Overview**
dashboard covering cluster throughput, connections,
partitions, request mix, consumer lag, and connector
activity. It is imported automatically when you save an
account, and you can open it from **Visualize Data > View
Dashboards** in the tile.
## Data freshness
Confluent Cloud reports metrics at a one-minute resolution,
and Oodle collects at the same one-minute granularity. New
data points typically appear in Oodle within a few minutes
of being recorded, in line with the Confluent Metrics API's
own publication schedule.
## Troubleshooting
| Issue | Resolution |
|-------|------------|
| Save fails with an authentication error | Confirm the API key has the **MetricsViewer** role and is scoped to the organization or environment, not a single cluster |
| No metrics after a few minutes | Verify the Cluster IDs (`lkc-...`) and Connector IDs (`lcc-...`) are correct and belong to the same organization as the API key |
| A metric is empty | The cluster or connector may not currently be producing that metric (for example, consumer lag appears once a consumer group is active) |
---
## Datadog
Oodle ingests metrics, logs, and traces (APM) from the
Datadog agent. This guide covers all supported signals
and configuration methods.
The quickest way to get started is through the
**Datadog integration tile** on the Settings page — it
generates the exact configuration for your instance.
## Supported signals
| Signal | Dual-ship env var | Single-ship env var |
|---------|------------------------------------------|--------------------------------|
| Metrics | `DD_ADDITIONAL_ENDPOINTS` | `DD_DD_URL` + `DD_API_KEY` |
| Logs | `DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS` | `DD_LOGS_CONFIG_DD_URL` |
| Traces | `DD_APM_ADDITIONAL_ENDPOINTS` | `DD_APM_DD_URL` |
:::note
The endpoint URL and API key for your instance are
shown in the Datadog tile on the
Settings → Integrations page.
:::
:::note
Wherever metrics are shipped to Oodle, set
`DD_USE_V3_API_SERIES_ENABLED=false` (agent config key
`use_v3_api.series.enabled: false`) so the Agent submits
series on the v2 metrics intake that Oodle accepts. This
setting is included in every metrics configuration below.
:::
## Dual-shipping (Datadog + Oodle)
Send data to both Datadog and Oodle simultaneously.
### Datadog Helm chart
Add the following to your Helm `values.yaml` for the
[Datadog Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/datadog):
```yaml
datadog:
env:
# Metrics
- name: DD_ADDITIONAL_ENDPOINTS
value: '{"": [""]}'
- name: DD_USE_V3_API_SERIES_ENABLED
value: 'false'
# Logs
- name: DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS
value: '[{"api_key": "", "Host": "", "Port": 443, "is_reliable": false}]'
- name: DD_LOGS_CONFIG_FORCE_USE_HTTP
value: 'true'
# APM / Traces
- name: DD_APM_ADDITIONAL_ENDPOINTS
value: '{"": [""]}'
clusterAgent:
env:
# Metrics (cluster agent also needs this)
- name: DD_ADDITIONAL_ENDPOINTS
value: '{"": [""]}'
- name: DD_USE_V3_API_SERIES_ENABLED
value: 'false'
```
Then install or upgrade:
```bash
helm repo add --force-update datadog \
https://helm.datadoghq.com && \
helm upgrade --install datadog-agent datadog/datadog \
--values values.yaml
```
### Datadog Operator
Apply the following `DatadogAgent` resource:
```yaml
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
name: datadog
spec:
override:
nodeAgent:
containers:
agent:
env:
# Metrics
- name: DD_ADDITIONAL_ENDPOINTS
value: '{"": [""]}'
- name: DD_USE_V3_API_SERIES_ENABLED
value: 'false'
# Logs
- name: DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS
value: '[{"api_key": "", "Host": "", "Port": 443, "is_reliable": false}]'
- name: DD_LOGS_CONFIG_FORCE_USE_HTTP
value: 'true'
trace-agent:
env:
# APM / Traces
- name: DD_APM_ADDITIONAL_ENDPOINTS
value: '{"": [""]}'
clusterAgent:
env:
# Metrics (cluster agent also needs this)
- name: DD_ADDITIONAL_ENDPOINTS
value: '{"": [""]}'
- name: DD_USE_V3_API_SERIES_ENABLED
value: 'false'
```
```bash
kubectl apply -f datadog-agent.yaml
```
### Datadog Agent YAML
Add the following to your
[datadog.yaml](https://github.com/Datadog/datadog-agent/blob/878600ef7a55c5ef0efb41ed0915f020cf7e3bd0/pkg/config/config_template.yaml#L33):
```yaml
# Metrics
additional_endpoints:
"":
-
use_v3_api:
series:
enabled: false
# APM / Traces
apm_config:
additional_endpoints:
"":
-
# Logs
logs_config:
force_use_http: true
additional_endpoints:
- api_key:
Host:
Port: 443
is_reliable: false
```
### Environment variables
If you are not using Helm, Operator, or YAML config,
set these environment variables on both the `agent` and
`cluster-agent` containers:
```bash
# Metrics
DD_ADDITIONAL_ENDPOINTS=\
'{"": [""]}'
DD_USE_V3_API_SERIES_ENABLED=false
# Logs
DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS=\
'[{"api_key": "", "Host": "", "Port": 443, "is_reliable": false}]'
DD_LOGS_CONFIG_FORCE_USE_HTTP='true'
# APM / Traces
DD_APM_ADDITIONAL_ENDPOINTS=\
'{"": [""]}'
```
## Single-shipping (Oodle only)
Stop sending data to Datadog and send only to Oodle.
### Datadog Helm chart
```yaml
datadog:
apiKey:
dd_url:
logs:
enabled: true
apm:
enabled: true
env:
# Metrics
- name: DD_USE_V3_API_SERIES_ENABLED
value: 'false'
# Logs
- name: DD_LOGS_CONFIG_DD_URL
value: ''
- name: DD_LOGS_CONFIG_FORCE_USE_HTTP
value: 'true'
# APM / Traces
- name: DD_APM_DD_URL
value: ''
```
Then install or upgrade:
```bash
helm repo add --force-update datadog \
https://helm.datadoghq.com && \
helm upgrade --install datadog-agent datadog/datadog \
--values values.yaml
```
### Datadog Operator
```yaml
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
name: datadog
spec:
global:
endpoint:
url:
credentials:
apiKey:
override:
nodeAgent:
containers:
agent:
env:
# Metrics
- name: DD_USE_V3_API_SERIES_ENABLED
value: 'false'
# Logs
- name: DD_LOGS_CONFIG_DD_URL
value: ''
- name: DD_LOGS_CONFIG_FORCE_USE_HTTP
value: 'true'
trace-agent:
env:
# APM / Traces
- name: DD_APM_DD_URL
value: ''
```
```bash
kubectl apply -f datadog-agent.yaml
```
### Datadog Agent YAML
```yaml
# Metrics
dd_url:
api_key:
use_v3_api:
series:
enabled: false
# APM / Traces
apm_config:
apm_dd_url:
# Logs
logs_config:
logs_dd_url:
force_use_http: true
```
### Environment variables
```bash
# Metrics
DD_DD_URL=
DD_API_KEY=
DD_USE_V3_API_SERIES_ENABLED=false
# Logs
DD_LOGS_CONFIG_DD_URL=''
DD_LOGS_CONFIG_FORCE_USE_HTTP='true'
# APM / Traces
DD_APM_DD_URL=''
```
## Amazon ECS
Run the Datadog Agent in [Amazon ECS](https://aws.amazon.com/ecs/)
and ship its metrics, logs, and traces to Oodle by setting the
same environment variables on the **Agent container**: the
injected sidecar on **Fargate**, or the Agent daemon task on
**EC2**. Both launch types use the official
[`terraform-aws-ecs-datadog`](https://github.com/DataDog/terraform-aws-ecs-datadog)
module (`ecs_fargate` / `ecs_ec2`), which exposes a
`dd_environment` input for exactly these variables.
:::tip
The fastest path is the **ECS integration tile** on the
Settings → Integrations page, which renders the exact
`environment` block for your instance. A runnable end-to-end
example for both launch types lives in the
[oodle-onboarding `datadog/ecs`](https://github.com/oodle-ai/oodle-onboarding/tree/main/datadog/ecs)
demo. For the full ECS guide (including log-only routers such
as FireLens / OpenTelemetry / CloudWatch), see the
[Amazon ECS guide](/ecs).
:::
### Dual-write (Datadog + Oodle)
Add Oodle as additional endpoints on the Agent container. This is
additive, so keep your existing `DD_API_KEY` / `DD_SITE`.
```json
"environment": [
{
"name": "DD_ADDITIONAL_ENDPOINTS",
"value": "{\"\": [\"\"]}"
},
{ "name": "DD_USE_V3_API_SERIES_ENABLED", "value": "false" },
{
"name": "DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS",
"value": "[{\"api_key\": \"\", \"Host\": \"\", \"Port\": 443, \"is_reliable\": false}]"
},
{ "name": "DD_LOGS_CONFIG_FORCE_USE_HTTP", "value": "true" },
{
"name": "DD_APM_ADDITIONAL_ENDPOINTS",
"value": "{\"\": [\"\"]}"
}
]
```
### Single-write (Oodle only)
Override the Agent's primary endpoints and authenticate with your
Oodle API key. No Datadog account is required.
```json
"environment": [
{ "name": "DD_API_KEY", "value": "" },
{ "name": "DD_DD_URL", "value": "" },
{ "name": "DD_USE_V3_API_SERIES_ENABLED", "value": "false" },
{ "name": "DD_APM_DD_URL", "value": "" },
{ "name": "DD_LOGS_CONFIG_LOGS_DD_URL", "value": ":443" },
{ "name": "DD_LOGS_CONFIG_FORCE_USE_HTTP", "value": "true" }
]
```
:::note
On the **EC2** launch type, also set
`DD_ORCHESTRATOR_EXPLORER_ENABLED=false`. Orchestrator Explorer
ships to Datadog's site-based process intake, which `DD_DD_URL`
cannot redirect to Oodle.
:::
---
## Google Cloud Platform
Oodle integrates with Google Cloud Platform (GCP) to
pull metrics from
[Cloud Monitoring](https://cloud.google.com/monitoring).
Oodle uses a service account with delegated access to
query your GCP project, discover compute resources, and
enrich metrics with resource labels and tags.
## Prerequisites
- A GCP project with billing enabled
- Permission to enable APIs, create service accounts,
and manage IAM bindings in the project
- An Oodle account (navigate to
to
start setup)
## Setup
1. Open the GCP integration page in Oodle
(),
or click **Settings** in the left sidebar and select
the **Google Cloud Platform** tile.
2. Click **Add project** (`+` button) to open the setup
drawer.
The setup drawer guides you through each step. Follow
the instructions in the drawer, or use the tabs below
for detailed guidance:
### 1. Provide GCP Project ID
Enter the ID of the GCP project you want to monitor
with Oodle. You can find your project ID in the
[GCP Console dashboard](https://console.cloud.google.com/).
### 2. Enable Required APIs
Ensure the following APIs are enabled for your project.
You can click each link in the setup drawer to check
and enable the API if needed:
| API | Purpose |
|-----|---------|
| [Cloud Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com) | Query Google Cloud metric data |
| [Compute Engine API](https://console.cloud.google.com/apis/library/compute.googleapis.com) | Discover compute instance data |
| [Cloud Asset API](https://console.cloud.google.com/apis/library/cloudasset.googleapis.com) | Request Google Cloud resources and link relevant labels as tags |
| [Cloud Resource Manager API](https://console.cloud.google.com/apis/library/cloudresourcemanager.googleapis.com) | Append metrics with the correct resources and tags |
### 3. Create a Service Account
[Create a service account](https://console.cloud.google.com/iam-admin/serviceaccounts/create)
that Oodle will use to access your GCP project.
Assign the following roles to the service account:
| Role | Access |
|------|--------|
| **Browser** | Read-only access to browse the hierarchy of a project |
| **Compute Viewer** | Read-only access to get and list Compute Engine resources |
| **Monitoring Viewer** | Read-only access to monitoring data in your Google Cloud environment |
| **Cloud Asset Viewer** | Read-only access to cloud assets metadata |
### 4. Provide GCP Service Account Email
Enter the email address of the service account you
created in the previous step (e.g.
`oodle-monitoring@YOUR_PROJECT_ID.iam.gserviceaccount.com`).
### 5. Add the Oodle Principal to Your Service Account
1. Open the **Principals with access** tab of your
service account in the
[IAM console](https://console.cloud.google.com/iam-admin/serviceaccounts).
2. Click **Grant access** to open the access drawer.
3. In the **New principals** field under **Add principals**,
paste the Oodle principal:
```
gcp-monitoring-integration@oodle-ai.iam.gserviceaccount.com
```
4. In the **Role** dropdown under **Assign roles**, select
**Service Account Token Creator**.
5. Click **Save**.
:::tip Troubleshooting: Organization Policy Restriction
If you see an error like *"one or more users named in
the policy do not belong to a permitted customer"*, your
organization has an `iam.allowedPolicyMemberDomains`
constraint. Add Oodle's Directory Customer ID
`C03pmpa0m` to your organization's allowed policy
member domains.
:::
### 6. Save and Wait
Click **Save** in the Oodle drawer. Allow up to
**5 minutes** for setup to complete and metrics to
start flowing.
### 1. Install gcloud CLI
Install the
[Google Cloud CLI](https://cloud.google.com/sdk/docs/install)
if you don't have it already.
### 2. Enable Required APIs
```bash
gcloud services enable \
monitoring.googleapis.com \
compute.googleapis.com \
cloudasset.googleapis.com \
cloudresourcemanager.googleapis.com \
--project=YOUR_PROJECT_ID
```
### 3. Create a Service Account
Create a service account and grant the required roles:
```bash
gcloud iam service-accounts create oodle-monitoring \
--display-name="Oodle Monitoring" \
--project=YOUR_PROJECT_ID
for role in roles/browser roles/compute.viewer \
roles/monitoring.viewer roles/cloudasset.viewer; do
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
--member="serviceAccount:oodle-monitoring@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
--role="$role"
done
```
### 4. Grant Oodle Principal Access
Grant the Oodle service account the
**Service Account Token Creator** role on your service
account:
```bash
gcloud iam service-accounts add-iam-policy-binding \
oodle-monitoring@YOUR_PROJECT_ID.iam.gserviceaccount.com \
--member="serviceAccount:gcp-monitoring-integration@oodle-ai.iam.gserviceaccount.com" \
--role="roles/iam.serviceAccountTokenCreator" \
--project=YOUR_PROJECT_ID
```
### 5. Enter Service Account Email and Save
Back in the Oodle setup drawer, enter the service
account email and click **Save**. Allow up to
**5 minutes** for setup to complete and metrics to
start flowing.
:::tip Troubleshooting: Organization Policy Restriction
If the `add-iam-policy-binding` command fails due to an
`iam.allowedPolicyMemberDomains` constraint, add
Oodle's Directory Customer ID `C03pmpa0m` to your
organization's allowed policy member domains.
:::
## Verification
Once setup is complete, verify the integration is
working:
1. Navigate to
and confirm the project status shows as connected.
2. Go to and search
for metrics prefixed with `gcp_` to confirm data is
flowing.
## Troubleshooting
| Issue | Resolution |
|-------|------------|
| No metrics appearing after 5 minutes | Verify all four APIs are enabled in the GCP project |
| Permission denied errors | Check that the service account has all four required roles |
| Token creation failures | Ensure the Oodle principal has the **Service Account Token Creator** role on your service account |
| Organization policy blocks external principals | Add Directory Customer ID `C03pmpa0m` to `iam.allowedPolicyMemberDomains` |
## How Oodle collects GCP metrics
### Polling interval
Oodle polls the Cloud Monitoring API on a recurring
interval - **every 5 minutes by default**. To adjust
this interval, reach out to Oodle Support.
A shorter polling interval increases the number of API
calls made to GCP, which affects your
[Cloud Monitoring API costs](https://cloud.google.com/stackdriver/pricing#monitoring-costs).
Note that reducing the polling interval below a metric's
native alignment interval (see below) will have no
impact - GCP will not have new data points to return
more frequently than the metric is emitted.
### GCP metric alignment intervals
Different GCP services emit metrics at different native
intervals. For example, Compute Engine CPU metrics are
reported every 60 seconds, while some managed-service
metrics (such as Cloud SQL or Cloud Storage) may only
update every 5 or 10 minutes. Oodle requests data
aligned to each metric's native interval so no data
points are missed or double-counted.
### Expected data delay
Because of the combination of Oodle's polling interval
and GCP's own metric publication delay, metrics
typically appear in Oodle **approximately 5 minutes**
after they are recorded in GCP. This delay is normal and
applies to all polling-based integrations.
### Custom metrics
If you are sending custom metrics to GCP and want them
available in Oodle without the polling delay, consider
writing those metrics directly to Oodle instead. See the
other [metrics integrations](/integrations/metrics) for
options on sending data to Oodle directly.
---
## Annotations
Oodle supports creating, updating, listing, and
deleting annotations via HTTP API. Use annotations to
mark deployment events, releases, configuration changes,
or any significant event on your dashboards as vertical
markers on time-series panels.
## Authentication
All requests require an API key passed in the
`X-API-KEY` header. You can find your API key on the
**Settings > Integrations** page in the Oodle UI.
## Create an annotation
```bash
curl -X POST \
https:///v1/api/instance//grafana/annotations \
-H "Content-Type: application/json" \
-H "X-API-KEY: " \
-d "{
\"text\": \"Deployed v1.2.3 to production\",
\"tags\": [\"deployment\", \"production\"],
\"time\": $(date +%s)000
}"
```
### Request body
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `text` | string | Yes | Description of the event |
| `tags` | string[] | No | Tags for filtering annotations |
| `time` | int64 | No | Start time in epoch milliseconds (defaults to now) |
| `timeEnd` | int64 | No | End time in epoch ms (for range annotations) |
| `dashboardId` | int64 | No | Scope to a specific dashboard |
| `panelId` | int64 | No | Scope to a specific panel |
### Response
```json
{
"id": 42,
"message": "Annotation added"
}
```
## List annotations
```bash
curl \
"https:///v1/api/instance//grafana/annotations?tags=deployment&limit=10" \
-H "X-API-KEY: "
```
### Query parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `from` | int64 | Start of time range (epoch ms) |
| `to` | int64 | End of time range (epoch ms) |
| `tags` | string | Filter by tag (repeat for multiple) |
| `dashboardId` | int64 | Filter by dashboard ID |
| `panelId` | int64 | Filter by panel ID |
| `limit` | int32 | Max annotations to return |
### Response
```json
[
{
"id": 42,
"dashboardId": 1,
"dashboardUID": "abc123",
"panelId": 0,
"time": 1718000000000,
"timeEnd": 0,
"text": "Deployed v1.2.3 to production",
"tags": ["deployment", "production"],
"created": 1718000000000,
"updated": 1718000000000
}
]
```
## Update an annotation
```bash
curl -X PUT \
"https:///v1/api/instance//grafana/annotations/" \
-H "Content-Type: application/json" \
-H "X-API-KEY: " \
-d '{
"text": "Deployed v1.2.4 (hotfix)",
"tags": ["deployment", "hotfix"]
}'
```
### Request body
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `text` | string | Yes | Updated description |
| `tags` | string[] | No | Updated tags |
| `time` | int64 | No | Updated start time (epoch ms) |
| `timeEnd` | int64 | No | Updated end time (epoch ms) |
### Response
```json
{
"message": "Annotation updated"
}
```
## Delete an annotation
```bash
curl -X DELETE \
"https:///v1/api/instance//grafana/annotations/" \
-H "X-API-KEY: "
```
### Response
```json
{
"message": "Annotation deleted"
}
```
## CI/CD integration
Add annotation creation to your CI/CD pipeline to
automatically mark deployments on dashboards:
### GitHub Actions
```yaml
- name: Create deployment annotation
run: |
curl -X POST \
"https://${{ secrets.OODLE_API_DOMAIN }}/v1/api/instance/${{ secrets.OODLE_INSTANCE }}/grafana/annotations" \
-H "Content-Type: application/json" \
-H "X-API-KEY: ${{ secrets.OODLE_API_KEY }}" \
-d "{
\"text\": \"Deployed ${{ github.sha }} to production\",
\"tags\": [\"deployment\", \"${{ github.ref_name }}\"],
\"time\": $(date +%s)000
}"
```
## Integration setup
You can find the API domain, instance ID, and API key
on the **Annotations** tile in the
[Integrations](/integrations) page of the Oodle UI.
---
## Grafana
To make migration from Grafana to Oodle easier, we support importing your
existing Grafana Dashboards, Alerts, and Datasources into Oodle automatically.
## Import Grafana Dashboards, Alerts, and Datasources
You can find the Grafana import steps by doing the following:
1. Login to the Oodle UI, then navigate to **Settings** page
2. Click on the Grafana tile
3. Choose an appropriate API key from the list on top of the drawer
You will be required to provide your Grafana instance URL and a Service
Account Token with an **Admin** role to read your Dashboards,
Datasources, and Alerts.
Please refer to the [Grafana documentation](https://grafana.com/docs/grafana/latest/administration/service-accounts/)
on how to create a Service Account Token with the Admin role.
Once you have provided the required information, click on `Import` to start the
Datasources into your Oodle account and will provide a report on the import
status. The status of the tile will change to `Imported` once this process is
complete.
### Dashboards
Your Grafana dashboards will be imported with their panels, queries, and folder
structure preserved. You can view and manage imported dashboards from the
**Dashboards** tab after import.
### Alerts
Alert rules configured in Grafana will be imported and can be managed from the
**Alerts** tab. Review the imported alerts to ensure they are configured
correctly for your Oodle environment.
### Datasources
Datasource configurations will be imported and mapped to equivalent Oodle
datasources where possible. You can review and manage imported datasources from
the **Data Sources** tab.
## Other Migration Options
Looking to migrate from other platforms? Check out these guides:
- [Alertmanager Alerts](/integrations/metrics/prometheus/alertmanager-alerts) -
Import alert rules and notification routes from Prometheus/Alertmanager
---
## Metrics
### Easily integrate Oodle with your existing systems.
---
---
## Kubernetes(Kubernetes)
Instrument your kubernetes cluster in one command!
Collect metrics, logs, kubernetes events and eBPF powered APM + Service Graph data from your Kubernetes cluster.
## Installation
1. Open `Settings` -> `Integrations` page in Oodle UI.
2. Choose `Kubernetes` integration tile.
3. Select the appropriate API key from the dropdown at the top of the drawer
4. Enter your Kubernetes cluster name and pick all the telemetry signals you want to collect
5. Based on the selected signals, a helm values file is generated, copy the values file
and run the helm install commands to complete the integration
### Uninstallation
To uninstall the helm chart, run the following command.
```bash
helm uninstall -n oodle-monitoring oodle-observability
```
## Architecture
Oodle's Helm chart adds lightweight, read-only collectors alongside your existing workloads. Your pods, nodes, and API server are untouched — Oodle only observes. Each signal is independently toggleable.
---
## Open Telemetry(Otel)
Oodle supports metrics ingestion via Open Telemetry.
## Zero-Code Instrumentation
For applications that don't have OpenTelemetry instrumentation built-in, you can use zero-code instrumentation to automatically capture telemetry data without modifying your application code.
### Supported Languages
- **[Java](java.md)** - Use the OpenTelemetry Java agent for automatic instrumentation
- **[Python](python.md)** - Use the `opentelemetry-instrument` CLI for automatic instrumentation
## Otel Collector Configuration
In the Oodle UI, navigate to the `Settings` page and click on `Connect` in the
`Open Telemetry` tile and copy the provided configuration snippet.
Oodle can ingest Open Telemetry metrics using the
[`otlphttp` exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlphttpexporter/README.md).
Add the `otlphttp` exporter from above in your Open Telemetry collector
configuration by adding an entry in the `exporters` section, and then adding
the exporter to the `service.pipelines.metrics.exporters` list.
```yaml title="otel-collector-config.yaml"
receivers:
otlp:
protocols:
http:
processors:
batch:
exporters:
# highlight-start
otlphttp/oodle:
endpoint: "https://"
headers:
X-OODLE-INSTANCE: ""
X-API-KEY: ""
# highlight-end
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
# highlight-next-line
exporters: [otlphttp/oodle]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/oodle]
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/oodle]
```
### Steps
You can find the Open Telemetry config changes by following these steps:
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Open Telemetry tile
3. Choose an appropriate API key from the list on top of the drawer
4. Ensure that you have selected the Metrics feature in Step 1.
Follow the steps in the drawer to complete the OpenTelemetry setup
### Otel To Prometheus Translation
Ingested Open Telemetry metrics are converted to Prometheus format.
#### Metric Name Normalization
Metric names are normalized by removing the
[characters that Prometheus does not support](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels)
with underscore (`_`) and dropping the redundant, leading and trailing
underscores. For example, `system.cpu.time` becomes `system_cpu_time`.
We also ensure that metric name does not start with a digit by prefixing it
with an underscore if necessary.
#### Attributes
Resource and metric attributes are converted to Prometheus labels with special
handling for following resource attributes:
- `service.name` is mapped to `job` label.
- `service.instance.id` is mapped to `instance` label.
#### Temporality
We currently do not support delta temporality for Sum, Histogram and Exponential
Histogram metric types.
## Visualize Metrics
You can visualize metrics in Oodle using the `Explore Metrics` tab. It supports
Grafana Builder and Code Editor to build PromQL queries. Please refer to
[PromQL Query docs](https://prometheus.io/docs/prometheus/latest/querying/basics/)
to learn more about PromQL.
---
## Java Zero-Code Instrumentation
Zero-code instrumentation with Java uses a Java agent JAR attached to any Java 8+ application. It dynamically injects bytecode to capture telemetry from many popular libraries and frameworks automatically, without requiring code changes.
## Overview
The OpenTelemetry Java agent can capture telemetry data at the "edges" of your application, such as:
- Inbound HTTP requests
- Outbound HTTP calls
- Database calls
- Messaging operations
- And many other popular frameworks and libraries
## Prerequisites
- Java 8 or higher
- Your Oodle metrics endpoint and API key
## Getting Started
### Step 1: Get Your Oodle Configuration
In the Oodle UI, navigate to the `Settings` page and click on `Connect` in the `Open Telemetry` tile and copy the provided configuration snippet which includes your metrics endpoint and API key
### Step 2: Download the Java Agent
Download the latest OpenTelemetry Java agent from the [official releases](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases):
```bash
wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
```
### Step 3: Configure Environment Variables
Set the following environment variables for your Java application:
```bash
# OpenTelemetry Configuration
export OTEL_SERVICE_NAME=
export OTEL_EXPORTER_OTLP_ENDPOINT=https://
export OTEL_EXPORTER_OTLP_HEADERS=X-API-KEY=,X-OODLE-INSTANCE=
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
# Optional: Service identification
export OTEL_RESOURCE_ATTRIBUTES=service.name=
```
### Step 4: Run Your Application with the Agent
Add the Java agent to your application startup command:
```bash
java -javaagent:opentelemetry-javaagent.jar -jar your-application.jar
```
## Configure using Integration tile
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Prometheus tile
3. Choose an appropriate API key from the list on top of the drawer
4. Click on Setup OpenTelemetry for Java
Follow the steps in the drawer to complete the OpenTelemetry setup for Java
## Docker Configuration
If you're running your Java application in Docker, add the agent to your Dockerfile:
```dockerfile
# Download the OpenTelemetry Java agent
RUN wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
# Set environment variables
ENV OTEL_EXPORTER_OTLP_ENDPOINT=https://
ENV OTEL_EXPORTER_OTLP_HEADERS=X-API-KEY=,X-OODLE-INSTANCE=
ENV OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
# Run your application with the agent
CMD ["java", "-javaagent:opentelemetry-javaagent.jar", "-jar", "your-application.jar"]
```
## Supported Instrumentations
The Java agent automatically instruments many popular libraries and frameworks. For a complete list, see the [OpenTelemetry Java Instrumentation documentation](https://opentelemetry.io/docs/instrumentation/java/automatic/).
## Configuration Options
### Suppressing Specific Instrumentations
You can disable specific instrumentations using environment variables:
```bash
export OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=false
export OTEL_INSTRUMENTATION_HTTP_ENABLED=false
export OTEL_INSTRUMENTATION_JDBC_ENABLED=false
```
## Performance Considerations
For detailed information about the performance impact and overhead of using the OpenTelemetry Java agent, please refer to the [official OpenTelemetry Java agent performance documentation](https://opentelemetry.io/docs/zero-code/java/agent/performance/).
## Troubleshooting
### Common Issues
1. **Agent not loading**: Ensure the `-javaagent` flag is correctly specified
2. **No metrics appearing**: Verify your Oodle endpoint and API key are correct
3. **Performance issues**: Check if you have too many instrumentations enabled
### Debug Mode
Enable debug logging to troubleshoot issues (not recommended for production as it generates a large volume of logs):
```bash
export OTEL_LOG_LEVEL=DEBUG
```
---
## Node.js Instrumentation
Use the [OpenTelemetry auto-instrumentation libraries](https://opentelemetry.io/docs/languages/js/libraries/#registration) to automatically capture telemetry from popular libraries and frameworks for your Node.js application - without modifying your application code.
## Overview
The OpenTelemetry Node.js auto-instrumentation can capture telemetry data at the "edges" of your application, such as:
- Inbound HTTP requests
- Outbound HTTP calls
- Database calls (MongoDB, PostgreSQL, MySQL, etc.)
- Express and other web framework operations
- And many other popular frameworks and libraries
## Prerequisites
- Node.js 14 or higher
- An active Oodle instance
## Getting Started
### Step 1: Get Your Oodle Configuration
In the Oodle UI, navigate to the `Settings` page and click on `Connect` in the `Open Telemetry` tile and copy the provided configuration snippet which includes your metrics endpoint and API key
### Step 2: Choose Your Setup Approach
The simplest way to get started - no instrumentation code required. Just install the package and configure via environment variables.
**Install the package:**
```bash
npm install @opentelemetry/auto-instrumentations-node
```
**Set environment variables:**
```bash
export OTEL_SERVICE_NAME=
export OTEL_EXPORTER_OTLP_ENDPOINT=
export OTEL_EXPORTER_OTLP_HEADERS="X-API-KEY="
export OTEL_METRICS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=none
```
**Run your application:**
```bash
node --require @opentelemetry/auto-instrumentations-node/register server.js
```
Or in `package.json`:
```json
{
"scripts": {
"start": "node --require @opentelemetry/auto-instrumentations-node/register server.js"
}
}
```
:::note Limitations
The zero-code approach uses default settings. For custom export intervals, selective instrumentation control, or advanced configuration, use the instrumentation.js file approach.
:::
For full control over export intervals, selective instrumentation, and custom resource attributes.
**Install the packages:**
```bash
npm install @opentelemetry/auto-instrumentations-node \
@opentelemetry/api \
@opentelemetry/sdk-node \
@opentelemetry/sdk-metrics \
@opentelemetry/sdk-trace-base \
@opentelemetry/exporter-metrics-otlp-http \
@opentelemetry/exporter-trace-otlp-http \
@opentelemetry/resources \
@opentelemetry/semantic-conventions
```
**Create an instrumentation file** named `instrumentation.js` in your project root:
```javascript
const { NodeSDK } = require('@opentelemetry/sdk-node');
const {
getNodeAutoInstrumentations,
} = require('@opentelemetry/auto-instrumentations-node');
const {
OTLPMetricExporter,
} = require('@opentelemetry/exporter-metrics-otlp-http');
const {
OTLPTraceExporter,
} = require('@opentelemetry/exporter-trace-otlp-http');
const {
PeriodicExportingMetricReader,
} = require('@opentelemetry/sdk-metrics');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { Resource } = require('@opentelemetry/resources');
const {
ATTR_SERVICE_NAME,
ATTR_DEPLOYMENT_ENVIRONMENT_NAME,
} = require('@opentelemetry/semantic-conventions');
// Read configuration from environment variables
const serviceName = process.env.OTEL_SERVICE_NAME ||
'my-service';
const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT ||
'http://localhost:4318';
const metricsEndpoint = process.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT ||
endpoint;
const tracesEndpoint = process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT ||
endpoint;
const apiKey = process.env.OTEL_EXPORTER_OTLP_HEADERS;
// Create resource with service name
const resource = new Resource({
[ATTR_SERVICE_NAME]: serviceName,
[ATTR_DEPLOYMENT_ENVIRONMENT_NAME]: 'production',
});
// Configure metric exporter
const metricExporter = new OTLPMetricExporter({
url: metricsEndpoint,
headers: apiKey ? { 'X-API-KEY': apiKey } : {},
});
// Configure trace exporter
const traceExporter = new OTLPTraceExporter({
url: tracesEndpoint,
headers: apiKey ? { 'X-API-KEY': apiKey } : {},
});
// Initialize SDK with auto-instrumentations
const sdk = new NodeSDK({
resource,
traceExporter,
metricReader: new PeriodicExportingMetricReader({
exporter: metricExporter,
exportIntervalMillis: 60000, // Export every 60 seconds
}),
spanProcessors: [
new BatchSpanProcessor(traceExporter),
],
instrumentations: [
getNodeAutoInstrumentations({
'@opentelemetry/instrumentation-http': { enabled: true },
'@opentelemetry/instrumentation-express': { enabled: true },
'@opentelemetry/instrumentation-dns': { enabled: true },
'@opentelemetry/instrumentation-net': { enabled: true },
}),
],
});
// Start the SDK
sdk.start();
console.log(
`OpenTelemetry instrumentation initialized for ${serviceName}`
);
console.log(` Traces endpoint: ${tracesEndpoint}`);
console.log(` Metrics endpoint: ${metricsEndpoint}`);
// Graceful shutdown
process.on('SIGTERM', () => {
sdk
.shutdown()
.then(
() => console.log('OpenTelemetry SDK shut down successfully'),
(err) =>
console.error('Error shutting down OpenTelemetry SDK', err)
)
.finally(() => process.exit(0));
});
```
**Configure environment variables:**
You can send telemetry data directly to Oodle or route it through an OpenTelemetry Collector for additional processing and routing capabilities.
Send metrics and traces directly to Oodle endpoints:
```bash
# OpenTelemetry Configuration
export OTEL_SERVICE_NAME=
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=
export OTEL_EXPORTER_OTLP_HEADERS=X-API-KEY=
```
**When to use**: Simple setups, single application, direct connection to Oodle.
Route telemetry through an OpenTelemetry Collector:
```bash
# OpenTelemetry Configuration
export OTEL_SERVICE_NAME=
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
```
Configure your OTEL Collector to forward to Oodle:
```yaml
# otel-collector-config.yaml
exporters:
otlphttp/oodle-metrics:
endpoint:
headers:
X-API-KEY:
otlphttp/oodle-traces:
endpoint:
headers:
X-API-KEY:
service:
pipelines:
metrics:
receivers: [otlp]
exporters: [otlphttp/oodle-metrics]
traces:
receivers: [otlp]
exporters: [otlphttp/oodle-traces]
```
**When to use**: Multiple services, need for data transformation, sampling, or routing to multiple backends.
**Run your application:**
Load the instrumentation file before your application code:
```bash
node --require ./instrumentation.js server.js
```
Update your `package.json` scripts:
```json
{
"scripts": {
"start": "node --require ./instrumentation.js server.js"
}
}
```
Then run:
```bash
npm start
```
Add the instrumentation setup to your Dockerfile:
```dockerfile
# Copy instrumentation file
COPY instrumentation.js .
# Set environment variables
ENV OTEL_SERVICE_NAME=
ENV OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=
ENV OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=
ENV OTEL_EXPORTER_OTLP_HEADERS=
# Run application with instrumentation
CMD ["node", "--require", "./instrumentation.js", "server.js"]
```
Or if using docker-compose.yml, override the command:
```yaml
services:
app:
image: your-app:latest
command: node --require ./instrumentation.js server.js
environment:
- OTEL_SERVICE_NAME=my-service
- OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://...
- OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://...
- OTEL_EXPORTER_OTLP_HEADERS=X-API-KEY=...
```
## Supported Instrumentations
The Node.js auto-instrumentation package supports many popular libraries and frameworks:
- **HTTP/HTTPS**: Express, Koa, Fastify, Hapi, Restify
- **Databases**: MongoDB, PostgreSQL, MySQL, Redis, Cassandra
- **Messaging**: Kafka, RabbitMQ, AWS SQS
- **Cloud Services**: AWS SDK, Google Cloud, Azure
For a complete list, see the [OpenTelemetry Node.js Instrumentation documentation](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/auto-instrumentations-node#supported-instrumentations).
## Configuration Options
### Enabling and Disabling Specific Instrumentations
Use environment variables to control which instrumentations are enabled:
```bash
# Disable specific instrumentations (comma-separated list)
export OTEL_NODE_DISABLED_INSTRUMENTATIONS="fs,dns"
# Or enable only specific instrumentations
export OTEL_NODE_ENABLED_INSTRUMENTATIONS="http,express,mongodb"
```
Configure instrumentations in your `instrumentation.js` file:
```javascript
instrumentations: [
getNodeAutoInstrumentations({
// Disable instrumentations you don't need
'@opentelemetry/instrumentation-fs': { enabled: false },
'@opentelemetry/instrumentation-dns': { enabled: false },
// Enable instrumentations you want
'@opentelemetry/instrumentation-http': { enabled: true },
'@opentelemetry/instrumentation-express': { enabled: true },
'@opentelemetry/instrumentation-mongodb': { enabled: true },
}),
]
```
### Adjusting Export Interval
```bash
# Export metrics every 60 seconds (in milliseconds)
export OTEL_METRIC_EXPORT_INTERVAL=60000
```
```javascript
metricReader: new PeriodicExportingMetricReader({
exporter: metricExporter,
exportIntervalMillis: 60000, // Export every 60 seconds
})
```
### Adding Custom Resource Attributes
```bash
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,service.namespace=my-namespace,deployment.region=us-west-2"
```
```javascript
const resource = new Resource({
[ATTR_SERVICE_NAME]: serviceName,
[ATTR_DEPLOYMENT_ENVIRONMENT_NAME]: 'production',
[ATTR_SERVICE_VERSION]: '1.0.0',
'service.namespace': 'my-namespace',
'deployment.region': 'us-west-2',
});
```
## Performance Considerations
The OpenTelemetry Node.js auto-instrumentation has minimal performance overhead:
- Typical CPU overhead: < 5%
- Memory overhead: < 50MB
- No noticeable impact on application latency
For production use:
- Start with conservative export intervals (30-60 seconds)
- Monitor your application's resource usage
- Disable unnecessary instrumentations
---
## Python Zero-Code Instrumentation
Zero-code instrumentation for Python uses the
`opentelemetry-instrument` CLI wrapper to automatically
capture telemetry from popular libraries and frameworks
— without modifying your application code.
## Overview
The OpenTelemetry Python zero-code instrumentation can
automatically capture telemetry at the "edges" of your
application, such as:
- Inbound HTTP requests (Django, Flask, FastAPI)
- Outbound HTTP calls (urllib3, requests, httpx)
- Database calls (psycopg, SQLAlchemy, pymysql)
- Messaging operations (Celery, Kafka, RabbitMQ)
- Logging (standard library `logging` module)
## Prerequisites
- Python 3.8 or higher
- `pip` package manager
- Your Oodle metrics endpoint and API key
## Getting Started
### Step 1: Get Your Oodle Configuration
In the Oodle UI, navigate to the `Settings` page and
click on `Connect` in the `Open Telemetry` tile and copy
the provided configuration snippet which includes your
metrics endpoint and API key.
### Step 2: Install the OpenTelemetry Packages
Install the core distro and OTLP exporter:
```bash
pip install opentelemetry-distro \
opentelemetry-exporter-otlp
```
Then run the bootstrap command to auto-detect and install
instrumentation packages for your application's
dependencies (e.g. Django, Flask, psycopg):
```bash
opentelemetry-bootstrap -a install
```
### Step 3: Configure Environment Variables
Set the following environment variables for your
application:
```bash
export OTEL_SERVICE_NAME=
export OTEL_EXPORTER_OTLP_ENDPOINT=
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_HEADERS="X-API-KEY="
# Auto-instrument Python logging
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
# Optional: resource attributes
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=,service.version=
```
### Step 4: Run Your Application
Wrap your normal startup command with
`opentelemetry-instrument`:
```bash
opentelemetry-instrument python manage.py runserver
```
This works with any Python entrypoint — Django, Flask,
FastAPI, Gunicorn, or a plain script:
```bash
# Django
opentelemetry-instrument python manage.py runserver
# Flask / FastAPI with Gunicorn
opentelemetry-instrument gunicorn app:app
# Plain script
opentelemetry-instrument python my_script.py
```
## Configure using Integration tile
1. Login to the Oodle UI, then navigate to Settings page
2. Click on the Prometheus tile
3. Choose an appropriate API key from the list on top of
the drawer
4. Click on Setup OpenTelemetry for Python
Follow the steps in the drawer to complete the
OpenTelemetry setup for Python.
## Docker Configuration
When running in Docker, use an
[OpenTelemetry Collector](https://opentelemetry.io/docs/collector/)
sidecar to receive telemetry from your application and
export it to Oodle.
### Dockerfile
```dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt \
&& opentelemetry-bootstrap -a install
COPY . .
CMD ["opentelemetry-instrument", \
"python", "manage.py", "runserver", \
"0.0.0.0:8000"]
```
Your `requirements.txt` should include:
```text
opentelemetry-distro>=0.48b0
opentelemetry-exporter-otlp>=1.27.0
```
### Docker Compose with OTel Collector
Use Docker Compose to run your app alongside an
OTel Collector that forwards telemetry to Oodle:
```yaml
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:0.96.0
command: ["--config=/etc/otel-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-config.yaml
environment:
- OODLE_API_KEY=${OODLE_API_KEY}
- OODLE_METRICS_ENDPOINT=${OODLE_METRICS_ENDPOINT}
- OODLE_TRACES_ENDPOINT=${OODLE_TRACES_ENDPOINT}
- OODLE_LOGS_ENDPOINT=${OODLE_LOGS_ENDPOINT}
ports:
- "4317:4317"
- "4318:4318"
app:
build: .
environment:
- OTEL_SERVICE_NAME=my-python-app
- OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
- OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
- OTEL_TRACES_EXPORTER=otlp
- OTEL_METRICS_EXPORTER=otlp
- OTEL_LOGS_EXPORTER=otlp
- OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
depends_on:
- otel-collector
```
### OTel Collector Config
Configure the collector to export to Oodle:
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 10s
send_batch_size: 1024
exporters:
otlphttp/oodle-metrics:
metrics_endpoint: ${env:OODLE_METRICS_ENDPOINT}
headers:
X-API-KEY: ${env:OODLE_API_KEY}
otlphttp/oodle-traces:
traces_endpoint: ${env:OODLE_TRACES_ENDPOINT}
headers:
X-API-KEY: ${env:OODLE_API_KEY}
otlphttp/oodle-logs:
logs_endpoint: ${env:OODLE_LOGS_ENDPOINT}
headers:
X-API-KEY: ${env:OODLE_API_KEY}
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/oodle-metrics]
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/oodle-traces]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/oodle-logs]
```
## Supported Instrumentations
The bootstrap command auto-detects and installs
instrumentation for many popular Python libraries.
For a complete list, see the
[OpenTelemetry Python Instrumentation documentation](https://opentelemetry.io/docs/zero-code/python/).
Common auto-detected libraries include:
| Library | Telemetry |
|---------|-----------|
| Django | Traces, metrics |
| Flask | Traces, metrics |
| FastAPI | Traces, metrics |
| psycopg / psycopg2 | Traces |
| SQLAlchemy | Traces |
| requests / urllib3 | Traces |
| Celery | Traces |
| logging | Logs |
## Configuration Options
### Suppressing Specific Instrumentations
Disable specific auto-instrumentations using environment
variables:
```bash
export OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=flask,urllib3
```
### Adjusting Metrics Export Interval
Control how frequently metrics are exported (default is
60 seconds):
```bash
export OTEL_METRIC_EXPORT_INTERVAL=10000
```
## Recording Exceptions in Traces
Zero-code instrumentation captures basic request-level
traces automatically, but application-level exceptions
caught in your own code won't appear in traces unless
you record them explicitly.
Use the OTel Trace SDK to attach error status and
exception details to the current span:
```python
# git-new-start
from opentelemetry import trace
from opentelemetry.trace import StatusCode
# git-new-end
try:
process(data)
except Exception as exc:
# git-new-start
span = trace.get_current_span()
span.set_status(StatusCode.ERROR, str(exc))
span.record_exception(exc)
# git-new-end
# ... handle exception ...
raise
```
- `set_status` marks the span as an error so it
surfaces in Oodle's trace error filters.
- `record_exception` adds the exception type, message,
and stack trace as a span event, visible in the
trace's **Events** tab.
This requires no additional dependencies — the
`opentelemetry-api` package is already installed as part
of the zero-code instrumentation setup.
The screenshot below shows how a recorded exception
appears in the Oodle Trace UI — the span is marked as
an error, and the full stack trace is visible in the
Logs panel on the right.
## Troubleshooting
### Common Issues
1. **No telemetry appearing**: Verify your Oodle endpoint
and API key are correct
2. **Missing library instrumentation**: Re-run
`opentelemetry-bootstrap -a install` after adding new
dependencies
3. **Logs not captured**: Ensure
`OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED`
is set to `true`
### Debug Mode
Enable debug logging to troubleshoot issues (not
recommended for production):
```bash
export OTEL_LOG_LEVEL=DEBUG
```
---
## Alertmanager Alerts
Oodle supports importing your existing Alertmanager alert rules and notification
routes, making migration from Prometheus/Alertmanager seamless.
## Import Methods
There are two ways to import alerts from Alertmanager into Oodle:
1. **Upload YAML files** - Upload your `prometheus.yaml` and
`alertmanager.yaml` configuration files directly
2. **Scrape from Prometheus** - Connect to your running Prometheus and
Alertmanager instances to scrape alerts directly
## Method 1: Import from YAML Files
This method allows you to upload your Prometheus and Alertmanager configuration
files directly to Oodle. This approach doesn't require network connectivity
between Oodle and your Prometheus/Alertmanager instances.
### Steps
1. Login to the Oodle UI, then navigate to **Settings** page
2. Click on the Alertmanager Alerts tile
3. Select the **Import Alerts from YAML files** tab
4. Upload your [`prometheus.yaml`](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
file containing your alert rules
5. Upload your [`alertmanager.yaml`](https://prometheus.io/docs/alerting/latest/configuration/)
file containing your notification routes
6. The import will start automatically once both files are uploaded
### What Gets Imported
- **Alert Rules**: All alert rules defined in your Prometheus configuration
- **Notification Routes**: Routing rules from your Alertmanager configuration
- **Notification Policies**: Receiver configurations mapped to Oodle notifiers
## Method 2: Scrape Alerts from Prometheus
This method connects directly to your Prometheus and Alertmanager instances to
scrape the alert configurations. This requires network connectivity between the
migration tool and your instances.
### Prerequisites
- Docker installed on your machine
([Install Docker](https://docs.docker.com/engine/install/))
- Network access to your Prometheus and Alertmanager instances
### Steps
1. Login to the Oodle UI, then navigate to **Settings** page
2. Search for **Alertmanager Alerts** and click **Import**
3. Choose an appropriate API key from the dropdown at the top of the drawer
4. Select the **Scrape Alerts from Prometheus** tab
5. Enter your Prometheus URL (e.g., `http://localhost:9090`)
6. Enter your Alertmanager URL (e.g., `http://localhost:9091`)
7. Copy and run the generated Docker command
### Docker Command
The migration tool runs as a Docker container. After entering your Prometheus
and Alertmanager URLs, a command will be generated:
```bash
docker run -it --pull=always --network host \
public.ecr.aws/oodle-ai/oodle/migrator:latest prometheus \
--prometheus-url http://host.docker.internal:9090 \
--alertmanager-url http://host.docker.internal:9091 \
--oodle-instance \
--oodle-api-key \
--oodle-endpoint
```
:::note
When using `localhost` URLs, they are automatically converted to
`host.docker.internal` to ensure the Docker container can reach your local
services.
:::
### Kubernetes Example
If Prometheus and Alertmanager are running in a Kubernetes cluster, you can use
port-forwarding to access them locally:
```bash
kubectl port-forward pods/ 9090 -n
kubectl port-forward pods/ 9093 -n
```
Then use `http://localhost:9090` and `http://localhost:9093` as the URLs for
Prometheus and Alertmanager respectively.
## Import Results
After the import completes, you'll see a status indicating success or failure.
The imported alerts will be available in the **Alerts** section of the Oodle UI.
You can manage your imported alerts:
- View all monitors in the [Monitors](/alerts/monitors) page
- Configure notification policies in
[Notification Policies](/alerts/notification-policies)
- Set up notifiers in [Notifiers](/alerts/notifiers)
## Other Migration Options
Looking to migrate from other platforms? Check out these guides:
- [Grafana Migration](/integrations/metrics/grafana) - Import Dashboards,
Alerts, and Datasources from Grafana
---
## Creating Custom Metrics
## Prerequisites: Understanding Prometheus Metric Types
Before implementing custom metrics, it's important to understand the [four core metric types](https://prometheus.io/docs/concepts/metric_types/) that Prometheus offers:
### Counter
A counter is a cumulative metric that represents a single monotonically increasing value. Counters can only increase or be reset to zero upon restart. Use counters for metrics like:
- Total number of requests processed
- Total number of errors
- Total number of tasks completed
**Do not use counters** for values that can decrease (like current number of running processes).
### Gauge
A gauge represents a single numerical value that can arbitrarily increase or decrease. Gauges are perfect for measuring current states like:
- Memory usage
- Active requests
- Queue size
- CPU utilization
### Histogram
A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It provides:
- Cumulative counters for observation buckets (`_bucket{le=""}`)
- Total sum of all observed values (`_sum`)
- Count of events observed (`_count`)
Histograms are ideal for calculating quantiles and Apdex scores.
### Summary
Similar to histograms, summaries sample observations but calculate quantiles over a sliding time window. A summary provides:
- Streaming φ-quantiles (0 ≤ φ ≤ 1) of observed events (`