Temporal

Send metrics, traces, and logs from Temporal to Oodle through a single OpenTelemetry Collector.

Temporal reports telemetry from two independent places, and a complete picture needs both:

• The platform, whichever way you run it. On Temporal Cloud that is the account OpenMetrics endpoint, covering namespace health, workflow and activity outcomes, task queue backlogs, service latencies, and rate limits. Self-hosted, it is the Temporal server's own Prometheus endpoint, covering service and persistence request rates, latencies, errors, and shard health. The collector scrapes whichever applies. No application change is needed.
• The SDK in your workers and clients emits its own metrics (worker slots, sticky cache, task schedule-to-start latency, workflow completions), plus traces and logs over OTLP. Trace context propagates through the Temporal server, so one trace covers a whole workflow across client, server, and worker. This is identical on Cloud and self-hosted.

The two platform sources are mutually exclusive: Cloud does not expose server internals, and a self-hosted server has no OpenMetrics endpoint. Follow the section matching your deployment, then instrument the SDK either way.

Prerequisites

• OODLE_INSTANCE: Your Oodle instance ID. Go to Settings icon -> API Keys page in your Oodle UI to find out. (Oodle UI links: ap1, us1)
• OODLE_API_KEY: Your Oodle API key for authentication. Go to Settings icon -> API Keys in your Oodle UI to choose an appropriate key. (Oodle UI links: ap1, us1)

Temporal Cloud: scrape the OpenMetrics endpoint

Temporal Cloud serves account-wide metrics from a single global endpoint, https://metrics.temporal.io/v1/metrics. It is Prometheus-compatible and authenticated with a bearer token.

note

The endpoint accepts only an API key belonging to a Service Account with the Metrics Read-Only role. A user API key returns 403, even for an Account Owner. mTLS no longer applies: Temporal disabled temporal cloud account metrics cert-ca create in favour of this endpoint.

Create the Service Account and its key. This requires Account Owner or Global Admin, and you can do it in the Temporal Cloud UI instead.

Install the Temporal CLI if it is not already present:

# macOS
brew install temporal

# Linux / WSL
curl -sSf https://temporal.download/cli.sh | sh

temporal cloud service-account create \
  --name oodle-metrics-scraper \
  --account-role metrics-read \
  --description "Scrapes Temporal Cloud OpenMetrics into Oodle"

umask 077
temporal cloud apikey create-for-service-account \
  --service-account-id <SERVICE_ACCOUNT_ID> \
  --display-name oodle-metrics-scraper \
  --expiry-duration 720h -o json \
  | jq -r .token > /etc/otel/temporal-metrics.key
chmod 600 /etc/otel/temporal-metrics.key

Check the key before configuring the collector. A new key takes up to a minute to become active, so retry a 403 once before assuming the role is wrong.

curl -s -H "Authorization: Bearer $(cat /etc/otel/temporal-metrics.key)" \
  https://metrics.temporal.io/v1/metrics | head

Add this scrape job to the collector config below. Temporal limits the endpoint to 180 requests per hour per account, so keep the interval at 30s (120 per hour) and run one collector per account.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "temporal-cloud"
          scrape_interval: 30s
          scrape_timeout: 30s
          honor_timestamps: true
          scheme: https
          metrics_path: /v1/metrics
          authorization:
            type: Bearer
            credentials_file: /etc/otel/temporal-metrics.key
          static_configs:
            - targets: ["metrics.temporal.io"]

Mount the key file into the collector alongside the config:

volumes:
  - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
  - ./temporal-metrics.key:/etc/otel/temporal-metrics.key:ro

Metrics arrive as temporal_cloud_v1_*. Temporal aggregates them over one-minute windows and publishes within three minutes, so the first points arrive after the first scrape.

warning

Every temporal_cloud_v1_* series is a gauge, and the *_count metrics are already expressed per second. Wrapping them in rate() returns nothing. Query sum(temporal_cloud_v1_service_request_count), not sum(rate(temporal_cloud_v1_service_request_count[5m])).

Self-hosted: enable the Temporal server metrics endpoint

Set PROMETHEUS_ENDPOINT on the Temporal server so it publishes metrics on port 8000. For a self-managed deployment, set the equivalent value in the server config. Skip this section on Temporal Cloud, which does not expose server internals.

services:
  temporal:
    image: temporalio/auto-setup:1.26.2
    environment:
      - PROMETHEUS_ENDPOINT=0.0.0.0:8000

OTel Collector configuration

Install the OpenTelemetry Collector Contrib distribution. The contrib build provides the prometheus receiver that scrapes the Temporal server.

One collector handles every signal: it scrapes the server, and it receives SDK traces, metrics, and logs over OTLP.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

  prometheus:
    config:
      scrape_configs:
        - job_name: "temporal-server"
          scrape_interval: 15s
          static_configs:
            - targets: ["temporal:8000"]

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  otlphttp/oodle:
    endpoint: "https://<OODLE_INSTANCE>-otlp.collector.oodle.ai"
    headers:
      "X-OODLE-INSTANCE": "<OODLE_INSTANCE>"
      "X-API-KEY": "<OODLE_API_KEY>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/oodle]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [otlphttp/oodle]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/oodle]

Run the collector with the config mounted and the OTLP ports published for your workers:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.114.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"
      - "4318:4318"
    environment:
      - OODLE_INSTANCE=<OODLE_INSTANCE>
      - OODLE_API_KEY=<OODLE_API_KEY>

The Temporal server metrics arrive as soon as the collector starts scraping.

Instrument the SDK

The examples below use the Python SDK. The Go, Java, and TypeScript SDKs expose the same two concepts: a runtime or telemetry option that carries metrics, and a tracing interceptor. See the Temporal observability docs.

Your application code holds no Oodle credentials. Only the collector does.

Install the packages

temporalio[opentelemetry]==1.11.0
opentelemetry-sdk==1.33.0
opentelemetry-api==1.33.0
opentelemetry-exporter-otlp-proto-grpc==1.33.0
opentelemetry-exporter-otlp-proto-http==1.33.0

Configure the providers

Traces and SDK metrics go to the collector over gRPC on port 4317. Logs go over HTTP on port 4318.

import logging

from opentelemetry import trace as otel_trace
from opentelemetry import _logs as otel_logs
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from temporalio.runtime import Runtime, TelemetryConfig, OpenTelemetryConfig

OTEL_ENDPOINT_GRPC = "http://otel-collector:4317"
OTEL_ENDPOINT_HTTP = "http://otel-collector:4318"
SERVICE_NAME = "temporal-worker"


def setup_opentelemetry() -> Runtime:
    resource = Resource.create({"service.name": SERVICE_NAME})

    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(endpoint=OTEL_ENDPOINT_GRPC, insecure=True)
        )
    )
    otel_trace.set_tracer_provider(tracer_provider)

    logger_provider = LoggerProvider(resource=resource)
    logger_provider.add_log_record_processor(
        BatchLogRecordProcessor(
            OTLPLogExporter(endpoint=f"{OTEL_ENDPOINT_HTTP}/v1/logs")
        )
    )
    otel_logs.set_logger_provider(logger_provider)

    root = logging.getLogger()
    root.setLevel(logging.INFO)
    root.addHandler(LoggingHandler(logger_provider=logger_provider))

    return Runtime(
        telemetry=TelemetryConfig(
            metrics=OpenTelemetryConfig(url=OTEL_ENDPOINT_GRPC)
        )
    )

Wire the runtime into the client

Pass the Runtime to Client.connect() to carry SDK metrics, and TracingInterceptor to carry traces. Give workflow starters the same setup with their own service.name.

from temporalio.client import Client
from temporalio.contrib.opentelemetry import TracingInterceptor
from temporalio.worker import Worker


async def main():
    runtime = setup_opentelemetry()

    client = await Client.connect(
        "temporal:7233",
        interceptors=[TracingInterceptor()],
        runtime=runtime,
    )

    worker = Worker(
        client,
        task_queue="my-task-queue",
        workflows=[OrderProcessingWorkflow],
        activities=[validate_order, process_payment],
    )
    await worker.run()

note

Two details decide whether the SDK signals arrive.

The Runtime must reach Client.connect(runtime=...). A runtime that is built and then left out of the call produces no SDK metrics.

Temporal replays workflow code, so its sandbox screens imports for determinism. Pass any OpenTelemetry import that workflow code reaches through the guard:

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from opentelemetry import trace as otel_trace

Dashboards

Oodle provides three prebuilt Temporal dashboards:

Dashboard	Applies to	Covers
Temporal Cloud Metrics	Cloud	Open workflows, service availability and latency, workflow and activity outcomes, task queue backlogs, Actions against namespace limits, schedules, replication lag
Temporal Server Metrics	Self-hosted	Service and persistence availability, request and error rates, latencies, pollers, shard rebalancing, workflow completion stats
Temporal SDK Metrics	Both	RPC requests and failures, workflow and activity latencies, task schedule-to-start, worker slots, sticky cache efficiency

To import them, open the Temporal tile on the Integrations page, go to the Dashboards tab, and choose Provision & View Dashboards. Oodle imports all three into a Temporal folder in Grafana and opens it. Running it again refreshes the dashboards in place. The dashboard that does not match your deployment simply stays empty. (Oodle UI links: ap1, us1)

Verify

Metrics, traces, and logs appear within about five minutes.

Signal	Where to look	What to expect
Cloud metrics	Metrics Explorer	temporal_cloud_v1_service_request_count, temporal_cloud_v1_approximate_backlog_count
Server metrics (self-hosted)	Metrics Explorer	service_requests, persistence_latency
SDK metrics	Metrics Explorer	Metrics prefixed temporal_, such as temporal_workflow_completed
Traces	Traces Explorer	Spans for service temporal-worker
Logs	Logs	Records for service temporal-worker, carrying trace context

If the platform metrics are absent while the SDK signals arrive, check that the collector image is the contrib build. Self-hosted, confirm it can reach the Temporal server on port 8000. On Cloud, check the collector log for the scrape status: a 403 means the key is not a Service Account key with the metrics-read role, and a 429 means the 180 requests per hour account limit was exceeded.

Demo

For a complete working example, see the Temporal demo in the oodle-onboarding repository.