Temporal

Send metrics, traces, and logs from Temporal to Oodle through a single OpenTelemetry Collector.

Temporal reports telemetry from two independent places, and a complete picture needs both:

• The Temporal server exposes a Prometheus endpoint covering service and persistence request rates, latencies, errors, shard health, and workflow completion counts. The collector scrapes it. No application change is needed.
• The SDK in your workers and clients emits its own metrics (worker slots, sticky cache, task schedule-to-start latency, workflow completions), plus traces and logs over OTLP. Trace context propagates through the Temporal server, so one trace covers a whole workflow across client, server, and worker.

Prerequisites

• OODLE_INSTANCE: Your Oodle instance ID. Go to Settings icon -> API Keys page in your Oodle UI to find out. (Oodle UI links: ap1, us1)
• OODLE_API_KEY: Your Oodle API key for authentication. Go to Settings icon -> API Keys in your Oodle UI to choose an appropriate key. (Oodle UI links: ap1, us1)

Enable the Temporal server metrics endpoint

Set PROMETHEUS_ENDPOINT on the Temporal server so it publishes metrics on port 8000. For a self-managed deployment, set the equivalent value in the server config.

services:
  temporal:
    image: temporalio/auto-setup:1.26.2
    environment:
      - PROMETHEUS_ENDPOINT=0.0.0.0:8000

OTel Collector configuration

Install the OpenTelemetry Collector Contrib distribution. The contrib build provides the prometheus receiver that scrapes the Temporal server.

One collector handles every signal: it scrapes the server, and it receives SDK traces, metrics, and logs over OTLP.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

  prometheus:
    config:
      scrape_configs:
        - job_name: "temporal-server"
          scrape_interval: 15s
          static_configs:
            - targets: ["temporal:8000"]

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  otlphttp/oodle:
    endpoint: "https://<OODLE_INSTANCE>-otlp.collector.oodle.ai"
    headers:
      "X-OODLE-INSTANCE": "<OODLE_INSTANCE>"
      "X-API-KEY": "<OODLE_API_KEY>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/oodle]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [otlphttp/oodle]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/oodle]

Run the collector with the config mounted and the OTLP ports published for your workers:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.114.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"
      - "4318:4318"
    environment:
      - OODLE_INSTANCE=<OODLE_INSTANCE>
      - OODLE_API_KEY=<OODLE_API_KEY>

The Temporal server metrics arrive as soon as the collector starts scraping.

Instrument the SDK

The examples below use the Python SDK. The Go, Java, and TypeScript SDKs expose the same two concepts: a runtime or telemetry option that carries metrics, and a tracing interceptor. See the Temporal observability docs.

Your application code holds no Oodle credentials. Only the collector does.

Install the packages

temporalio[opentelemetry]==1.11.0
opentelemetry-sdk==1.33.0
opentelemetry-api==1.33.0
opentelemetry-exporter-otlp-proto-grpc==1.33.0
opentelemetry-exporter-otlp-proto-http==1.33.0

Configure the providers

Traces and SDK metrics go to the collector over gRPC on port 4317. Logs go over HTTP on port 4318.

import logging

from opentelemetry import trace as otel_trace
from opentelemetry import _logs as otel_logs
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from temporalio.runtime import Runtime, TelemetryConfig, OpenTelemetryConfig

OTEL_ENDPOINT_GRPC = "http://otel-collector:4317"
OTEL_ENDPOINT_HTTP = "http://otel-collector:4318"
SERVICE_NAME = "temporal-worker"


def setup_opentelemetry() -> Runtime:
    resource = Resource.create({"service.name": SERVICE_NAME})

    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(endpoint=OTEL_ENDPOINT_GRPC, insecure=True)
        )
    )
    otel_trace.set_tracer_provider(tracer_provider)

    logger_provider = LoggerProvider(resource=resource)
    logger_provider.add_log_record_processor(
        BatchLogRecordProcessor(
            OTLPLogExporter(endpoint=f"{OTEL_ENDPOINT_HTTP}/v1/logs")
        )
    )
    otel_logs.set_logger_provider(logger_provider)

    root = logging.getLogger()
    root.setLevel(logging.INFO)
    root.addHandler(LoggingHandler(logger_provider=logger_provider))

    return Runtime(
        telemetry=TelemetryConfig(
            metrics=OpenTelemetryConfig(url=OTEL_ENDPOINT_GRPC)
        )
    )

Wire the runtime into the client

Pass the Runtime to Client.connect() to carry SDK metrics, and TracingInterceptor to carry traces. Give workflow starters the same setup with their own service.name.

from temporalio.client import Client
from temporalio.contrib.opentelemetry import TracingInterceptor
from temporalio.worker import Worker


async def main():
    runtime = setup_opentelemetry()

    client = await Client.connect(
        "temporal:7233",
        interceptors=[TracingInterceptor()],
        runtime=runtime,
    )

    worker = Worker(
        client,
        task_queue="my-task-queue",
        workflows=[OrderProcessingWorkflow],
        activities=[validate_order, process_payment],
    )
    await worker.run()

note

Two details decide whether the SDK signals arrive.

The Runtime must reach Client.connect(runtime=...). A runtime that is built and then left out of the call produces no SDK metrics.

Temporal replays workflow code, so its sandbox screens imports for determinism. Pass any OpenTelemetry import that workflow code reaches through the guard:

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from opentelemetry import trace as otel_trace

Dashboards

Oodle provides two prebuilt Temporal dashboards:

Dashboard	Covers
Temporal Server Metrics	Service and persistence availability, request and error rates, latencies, pollers, shard rebalancing, workflow completion stats
Temporal SDK Metrics	RPC requests and failures, workflow and activity latencies, task schedule-to-start, worker slots, sticky cache efficiency

To import them, open the Temporal tile on the Integrations page, go to the Dashboards tab, and choose Provision & View Dashboards. Oodle imports both dashboards into a Temporal folder in Grafana and opens it. Running it again refreshes the dashboards in place. (Oodle UI links: ap1, us1)

Verify

Metrics, traces, and logs appear within about five minutes.

Signal	Where to look	What to expect
Server metrics	Metrics Explorer	service_requests, persistence_latency
SDK metrics	Metrics Explorer	Metrics prefixed temporal_, such as temporal_workflow_completed
Traces	Traces Explorer	Spans for service temporal-worker
Logs	Logs	Records for service temporal-worker, carrying trace context

If the server metrics are absent while the SDK signals arrive, check that the collector image is the contrib build and that it can reach the Temporal server on port 8000.

Demo

For a complete working example, see the Temporal demo in the oodle-onboarding repository.