CockroachDB

Send metrics from CockroachDB to Oodle through an OpenTelemetry Collector.

Every CockroachDB node serves Prometheus-format metrics at /_status/vars on its HTTP port. The endpoint is on by default, so the cluster itself needs no configuration change. The collector scrapes each node and forwards the metrics to Oodle over OTLP.

Oodle ships four CockroachDB dashboards and twelve recommended monitors, both provisioned in one click from the integration tile.

Prerequisites

• A running CockroachDB cluster, with each node's HTTP port reachable from the collector
• OODLE_INSTANCE: Your Oodle instance ID. Go to the Settings icon -> API Keys page in your Oodle UI to find it. (Oodle UI links: ap1, us1)
• OODLE_API_KEY: Your Oodle API key for authentication. Go to the Settings icon -> API Keys in your Oodle UI to choose an appropriate key. (Oodle UI links: ap1, us1)
• An Oodle account (navigate to ap1, us1 to start setup)

OTel Collector configuration

Install the OpenTelemetry Collector Contrib distribution. The contrib build provides the prometheus receiver that scrapes the CockroachDB nodes. The core build does not include it.

Save the following as otel-collector-config.yaml. List one target per node, and set crdb_cluster to a name for the cluster.

receivers:
  # Every CockroachDB node serves Prometheus-format metrics at
  # /_status/vars on its HTTP port.
  prometheus:
    config:
      scrape_configs:
        - job_name: "cockroachdb"
          scrape_interval: 15s
          metrics_path: "/_status/vars"
          static_configs:
            - targets: ["roach1:8080"]
              labels:
                crdb_node: "roach1"
            - targets: ["roach2:8080"]
              labels:
                crdb_node: "roach2"
            - targets: ["roach3:8080"]
              labels:
                crdb_node: "roach3"
          relabel_configs:
            # The dashboards and monitors group by crdb_cluster, so several
            # clusters can share one Oodle instance without their series
            # colliding.
            - target_label: "crdb_cluster"
              replacement: "my-cluster"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  # Every rewrite below is CockroachDB-specific, so each one is guarded on the
  # scrape job. The Prometheus receiver maps job_name to the service.name
  # resource attribute, which is also what Oodle renders as the job label.
  # Without the guard, any other scrape job in this pipeline would have its
  # metrics renamed, its job label overwritten, and its scope stripped.
  transform/crdb:
    metric_statements:
      # CockroachDB uses bare metric names such as ranges, replicas, and
      # capacity_used. Namespace them with crdb_ so they cannot collide with
      # other sources in the same Oodle instance. up and scrape_* keep their
      # standard names: the receiver reports those about the scrape, not
      # CockroachDB, and the Node Down monitor reads up.
      - context: metric
        conditions:
          - resource.attributes["service.name"] == "cockroachdb"
        statements:
          - set(name, Concat(["crdb_", name], ""))
              where not IsMatch(name, "^crdb_")
              and not IsMatch(name, "^(up|scrape_.*)$")

      # Oodle turns every OTLP resource attribute into a metric label. The
      # Prometheus receiver attaches scrape-target metadata as resource
      # attributes, all of which restate "instance", so drop them rather than
      # carry six extra labels on every series. service.name is left alone:
      # the receiver already set it to the job name.
      - context: resource
        conditions:
          - attributes["service.name"] == "cockroachdb"
        statements:
          - set(attributes["db.system"], "cockroachdb")
          - delete_key(attributes, "net.host.name")
          - delete_key(attributes, "net.host.port")
          - delete_key(attributes, "server.address")
          - delete_key(attributes, "server.port")
          - delete_key(attributes, "http.scheme")
          - delete_key(attributes, "url.scheme")

      # Same idea for the instrumentation scope: the receiver's own Go package
      # path is not useful as a label.
      - context: scope
        conditions:
          - resource.attributes["service.name"] == "cockroachdb"
        statements:
          - set(name, "")
          - set(version, "")

  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlphttp/oodle:
    endpoint: "https://${env:OODLE_INSTANCE}-otlp.collector.oodle.ai"
    headers:
      "X-OODLE-INSTANCE": "${env:OODLE_INSTANCE}"
      "X-API-KEY": "${env:OODLE_API_KEY}"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter, transform/crdb, batch]
      exporters: [otlphttp/oodle]

Metric names

The bundled dashboards and monitors query the crdb_ prefixed names, so the transform/crdb processor must stay in the pipeline. The raw exposition on /_status/vars and the CockroachDB DB Console keep the unprefixed names.

Sharing the pipeline with other scrape jobs

Each statement is conditioned on service.name == "cockroachdb", which the Prometheus receiver sets from job_name. Other scrape jobs in the same pipeline pass through untouched: they keep their own metric names, their own job label, and their own instrumentation scope.

Labels

Two labels identify a series, and both the dashboards and the monitors depend on them:

Label	Source	Purpose
crdb_cluster	relabel_configs in the scrape job	Groups the nodes of one cluster. Give each cluster its own value.
crdb_node	Per-target labels in the scrape job	Identifies one node. Use the node's host name.
job	job_name in the scrape job	Must stay cockroachdb. The dashboards and monitors filter on it.

Run the collector

services:
  otel-collector:
    # The contrib distribution is required: the core image has no
    # prometheus receiver.
    image: otel/opentelemetry-collector-contrib:0.114.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    environment:
      - OODLE_INSTANCE=${OODLE_INSTANCE}
      - OODLE_API_KEY=${OODLE_API_KEY}

Metric volume

CockroachDB publishes about 2,000 data points per node per scrape, and the integration ships all of them. Every family is therefore in Oodle when you go looking for it, and all four dashboards and all twelve monitors resolve without further configuration.

scrape_interval in the scrape job is the lever if that volume matters.

Verify

1. Go to ap1, us1 and search for crdb_liveness_livenodes. Its value is the number of live nodes in the cluster.
2. Confirm the labels are present: the series carries job="cockroachdb", a crdb_cluster value, and one crdb_node value per node.
3. Open the CockroachDB tile (ap1, us1) and provision the dashboards and monitors.

Dashboards

Open the Dashboards tab in the CockroachDB tile and click View Dashboards to import all four. Provisioning overwrites by name, so it is safe to repeat.

Dashboard	Covers
CockroachDB Runtime	Live node count, CPU, memory, uptime, goroutines, clock offset
CockroachDB SQL	Query rates by statement type, connections, service and execution latency, transactions, full scans
CockroachDB Storage	Capacity used and available, live bytes, read amplification, compactions, file descriptors
CockroachDB Replication	Ranges, replicas, leaseholders, under-replicated and unavailable ranges, snapshots, rebalancing

Each dashboard has a Cluster and a Node variable, so one dashboard serves every cluster that reports to the instance.

Monitors

Open the Monitors tab in the tile and click Provision Recommended Monitors. Monitors are matched by name, so your edits are kept and provisioning again restores only the ones you removed.

Monitor	Severity	Fires when
Node Down	Critical	A node stops answering scrapes for 15 minutes
Unavailable Ranges	Critical	A range has no quorum
Store Disk Low	Critical	A store has less than 15% of its capacity available
Cluster Disk Low	Critical	The cluster has less than 20% of its capacity available
Clock Offset Near Max	Critical	A node's clock drifts toward the point where it removes itself from the cluster
Liveness Mismatch	Warning	Fewer nodes report as live than answer scrapes
Underreplicated Ranges	Warning	A range has fewer replicas than its zone configuration asks for
Instance Flapping	Warning	A node restarts more than once in ten minutes
Version Mismatch	Warning	The cluster runs more than one CockroachDB version for four hours
High Open FD Count	Warning	A node uses more than 80% of its file descriptor limit
Slow Raft Requests	Warning	Raft proposals stay in flight longer than expected
SQL Latency High	Warning	The 99th percentile of SQL service latency passes one second

Metrics collected

Every metric CockroachDB publishes at /_status/vars reaches Oodle. These are the families the dashboards and monitors use.

Metric	Description
crdb_liveness_livenodes	Nodes the cluster considers live
crdb_sys_uptime	Seconds since the node process started
crdb_sys_cpu_combined_percent_normalized	Node CPU use, normalized across cores
crdb_sys_rss	Resident memory of the node process
crdb_clock_offset_meannanos	Mean clock offset from the other nodes
crdb_sql_conns	Open SQL connections
crdb_sql_query_count	SQL statements run
crdb_sql_select_count, crdb_sql_insert_count, crdb_sql_update_count, crdb_sql_delete_count	SQL statements by type
crdb_sql_failure_count	SQL statements that returned an error
crdb_sql_full_scan_count	Statements that ran a full table scan
crdb_sql_service_latency_bucket, crdb_sql_exec_latency_bucket	SQL latency histograms
crdb_txn_commits, crdb_txn_aborts	Transaction outcomes
crdb_capacity, crdb_capacity_used, crdb_capacity_available	Store capacity
crdb_livebytes	Bytes of live data
crdb_rocksdb_read_amplification	Read amplification of the storage engine
crdb_ranges, crdb_ranges_underreplicated, crdb_ranges_unavailable	Range health
crdb_replicas, crdb_replicas_leaders, crdb_replicas_leaseholders	Replica and leaseholder counts
crdb_sys_fd_open, crdb_sys_fd_softlimit	Open file descriptors against the soft limit

Troubleshooting

Issue	Resolution
The collector logs unknown type "prometheus"	Use the otel/opentelemetry-collector-contrib image. The core build has no prometheus receiver.
No metrics arrive	Confirm the collector reaches each node's HTTP port (8080 by default) and that /_status/vars answers with curl.
Dashboard panels are empty	Confirm the scrape job is named cockroachdb and that crdb_cluster is set. The panels filter on both.
Series carry extra labels	Confirm the transform/crdb processor is in the metrics pipeline.
Another job's metrics are named crdb_* or labelled job="cockroachdb"	Confirm every statement group in transform/crdb keeps its service.name == "cockroachdb" condition.
A monitor never fires	Confirm the metric it reads is present in ap1, us1, and that the collector is not dropping it.

Demo

A runnable three-node cluster with the collector and a workload generator is in oodle-onboarding.