CockroachDB

Send metrics from CockroachDB to Oodle through an OpenTelemetry Collector.

Every CockroachDB node serves Prometheus-format metrics at /_status/vars on its HTTP port. The endpoint is on by default, so the cluster itself needs no configuration change. The collector scrapes each node and forwards the metrics to Oodle over OTLP.

Oodle ships four CockroachDB dashboards and twelve recommended monitors, both provisioned in one click from the integration tile.

Prerequisites

A running CockroachDB cluster, with each node's HTTP port reachable from the collector
OODLE_INSTANCE: Your Oodle instance ID. Go to the Settings icon -> API Keys page in your Oodle UI to find it. (Oodle UI links: ap1, us1)
OODLE_API_KEY: Your Oodle API key for authentication. Go to the Settings icon -> API Keys in your Oodle UI to choose an appropriate key. (Oodle UI links: ap1, us1)
An Oodle account (navigate to ap1, us1 to start setup)

OTel Collector configuration

Install the OpenTelemetry Collector Contrib distribution. The contrib build provides the prometheus receiver that scrapes the CockroachDB nodes. The core build does not include it.

Save the following as otel-collector-config.yaml. List one target per node, and set crdb_cluster to a name for the cluster.

receivers:
  # Every CockroachDB node serves Prometheus-format metrics at
  # /_status/vars on its HTTP port.
  prometheus:
    config:
      scrape_configs:
        - job_name: "cockroachdb"
          scrape_interval: 15s
          metrics_path: "/_status/vars"
          static_configs:
            - targets: ["roach1:8080"]
              labels:
                crdb_node: "roach1"
            - targets: ["roach2:8080"]
              labels:
                crdb_node: "roach2"
            - targets: ["roach3:8080"]
              labels:
                crdb_node: "roach3"
          relabel_configs:
            # The dashboards and monitors group by crdb_cluster, so several
            # clusters can share one Oodle instance without their series
            # colliding.
            - target_label: "crdb_cluster"
              replacement: "my-cluster"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  # Oodle turns every OTLP resource attribute into a metric label. The
  # Prometheus receiver attaches scrape-target metadata as resource
  # attributes, all of which restate "instance", so drop them rather than
  # carry six extra labels on every series.
  resource:
    attributes:
      - key: service.name
        value: "cockroachdb"
        action: upsert
      - key: db.system
        value: "cockroachdb"
        action: upsert
      - key: net.host.name
        action: delete
      - key: net.host.port
        action: delete
      - key: server.address
        action: delete
      - key: server.port
        action: delete
      - key: http.scheme
        action: delete
      - key: url.scheme
        action: delete

  # Same idea for the instrumentation scope: the receiver's own Go package
  # path is not useful as a label.
  transform/drop_scope:
    metric_statements:
      - context: scope
        statements:
          - set(name, "")
          - set(version, "")

  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlphttp/oodle:
    endpoint: "https://${env:OODLE_INSTANCE}-otlp.collector.oodle.ai"
    headers:
      "X-OODLE-INSTANCE": "${env:OODLE_INSTANCE}"
      "X-API-KEY": "${env:OODLE_API_KEY}"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter, resource, transform/drop_scope, batch]
      exporters: [otlphttp/oodle]

Labels

Two labels identify a series, and both the dashboards and the monitors depend on them:

Label	Source	Purpose
`crdb_cluster`	`relabel_configs` in the scrape job	Groups the nodes of one cluster. Give each cluster its own value.
`crdb_node`	Per-target `labels` in the scrape job	Identifies one node. Use the node's host name.
`job`	`job_name` in the scrape job	Must stay `cockroachdb`. The dashboards and monitors filter on it.

Run the collector

services:
  otel-collector:
    # The contrib distribution is required: the core image has no
    # prometheus receiver.
    image: otel/opentelemetry-collector-contrib:0.114.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    environment:
      - OODLE_INSTANCE=${OODLE_INSTANCE}
      - OODLE_API_KEY=${OODLE_API_KEY}

Send fewer metrics

CockroachDB publishes about 2,000 data points per node per scrape. To ship only the families the bundled dashboards and monitors use, add a filter processor. This brings the volume down to about 28 data points per node.

processors:
  filter/crdb:
    metrics:
      include:
        match_type: regexp
        metric_names:
          - "^sql_(query|select|insert|update|delete|failure|full_scan)_count$"
          - "^sql_conns$"
          - "^sql_(exec|service)_latency$"
          - "^liveness_livenodes$"
          - "^ranges(_underreplicated|_unavailable)?$"
          - "^replicas(_leaders|_leaseholders)?$"
          - "^capacity(_used|_available)?$"
          - "^livebytes$"
          - "^rocksdb_read_amplification$"
          - "^clock_offset_meannanos$"
          - "^sys_(cpu_combined_percent_normalized|rss|uptime)$"
          - "^txn_(commits|aborts)$"

service:
  pipelines:
    metrics:
      processors: [memory_limiter, resource, filter/crdb, transform/drop_scope, batch]

Match a histogram by its bare name (sql_service_latency), not by the Prometheus-style sql_service_latency_bucket. By the time the filter runs, the receiver has folded the _bucket, _sum, and _count lines into one OTLP histogram. Oodle expands it again at query time.

Verify

Go to ap1, us1 and search for liveness_livenodes. Its value is the number of live nodes in the cluster.
Confirm the labels are present: the series carries job="cockroachdb", a crdb_cluster value, and one crdb_node value per node.
Open the CockroachDB tile (ap1, us1) and provision the dashboards and monitors.

Dashboards

Open the Dashboards tab in the CockroachDB tile and click View Dashboards to import all four. Provisioning overwrites by name, so it is safe to repeat.

Dashboard	Covers
CockroachDB Runtime	Live node count, CPU, memory, uptime, goroutines, clock offset
CockroachDB SQL	Query rates by statement type, connections, service and execution latency, transactions, full scans
CockroachDB Storage	Capacity used and available, live bytes, read amplification, compactions, file descriptors
CockroachDB Replication	Ranges, replicas, leaseholders, under-replicated and unavailable ranges, snapshots, rebalancing

Each dashboard has a Cluster and a Node variable, so one dashboard serves every cluster that reports to the instance.

Monitors

Open the Monitors tab in the tile and click Provision Recommended Monitors. Monitors are matched by name, so your edits are kept and provisioning again restores only the ones you removed.

Monitor	Severity	Fires when
Node Down	Critical	A node stops answering scrapes for 15 minutes
Unavailable Ranges	Critical	A range has no quorum
Store Disk Low	Critical	A store has less than 15% of its capacity available
Cluster Disk Low	Critical	The cluster has less than 20% of its capacity available
Clock Offset Near Max	Critical	A node's clock drifts toward the point where it removes itself from the cluster
Liveness Mismatch	Warning	Fewer nodes report as live than answer scrapes
Underreplicated Ranges	Warning	A range has fewer replicas than its zone configuration asks for
Instance Flapping	Warning	A node restarts more than once in ten minutes
Version Mismatch	Warning	The cluster runs more than one CockroachDB version for four hours
High Open FD Count	Warning	A node uses more than 80% of its file descriptor limit
Slow Raft Requests	Warning	Raft proposals stay in flight longer than expected
SQL Latency High	Warning	The 99th percentile of SQL service latency passes one second

Metrics collected

Every metric CockroachDB publishes at /_status/vars reaches Oodle unless you add the filter above. These are the families the dashboards and monitors use.

Metric	Description
`liveness_livenodes`	Nodes the cluster considers live
`sys_uptime`	Seconds since the node process started
`sys_cpu_combined_percent_normalized`	Node CPU use, normalized across cores
`sys_rss`	Resident memory of the node process
`clock_offset_meannanos`	Mean clock offset from the other nodes
`sql_conns`	Open SQL connections
`sql_query_count`	SQL statements run
`sql_select_count`, `sql_insert_count`, `sql_update_count`, `sql_delete_count`	SQL statements by type
`sql_failure_count`	SQL statements that returned an error
`sql_full_scan_count`	Statements that ran a full table scan
`sql_service_latency`, `sql_exec_latency`	SQL latency histograms
`txn_commits`, `txn_aborts`	Transaction outcomes
`capacity`, `capacity_used`, `capacity_available`	Store capacity
`livebytes`	Bytes of live data
`rocksdb_read_amplification`	Read amplification of the storage engine
`ranges`, `ranges_underreplicated`, `ranges_unavailable`	Range health
`replicas`, `replicas_leaders`, `replicas_leaseholders`	Replica and leaseholder counts
`sys_fd_open`, `sys_fd_softlimit`	Open file descriptors against the soft limit

Troubleshooting

Issue	Resolution
The collector logs `unknown type "prometheus"`	Use the `otel/opentelemetry-collector-contrib` image. The core build has no `prometheus` receiver.
No metrics arrive	Confirm the collector reaches each node's HTTP port (8080 by default) and that `/_status/vars` answers with `curl`.
Dashboard panels are empty	Confirm the scrape job is named `cockroachdb` and that `crdb_cluster` is set. The panels filter on both.
Series carry extra labels	Confirm the `resource` and `transform/drop_scope` processors are in the metrics pipeline.
A monitor never fires	Confirm the metric it reads is present. Metrics outside the `filter/crdb` list are dropped when you enable that processor.

Demo

A runnable three-node cluster with the collector and a workload generator is in oodle-onboarding.

Support

If you need assistance or have any questions, please reach out to us through:

Email at [email protected]

Prerequisites​

OTel Collector configuration​

Labels​

Run the collector​

Send fewer metrics​

Verify​

Dashboards​

Monitors​

Metrics collected​

Troubleshooting​

Demo​