Crusoe

Send metrics and logs from Crusoe Cloud virtual machines (VMs) and Crusoe Managed Kubernetes (CMK) clusters to Oodle with Crusoe Telemetry Conduit.

Telemetry Conduit exports what the Crusoe Watch Agent collects: GPU (DCGM), CPU, memory, disk, network, InfiniBand and NVLink metrics, and node and CMK control plane logs. It gives two ways to send data to Oodle:

Method	Data	What you run	Availability
OpenTelemetry streaming	Metrics and logs	Nothing: Crusoe pushes to Oodle over OTLP	Limited availability. Ask Crusoe to turn it on.
Scrape endpoint	Metrics	An OpenTelemetry Collector that scrapes Crusoe and sends to Oodle	All Crusoe projects

Prerequisites

• The Crusoe Watch Agent on your VMs or CMK nodes. Telemetry Conduit exports only what the agent collects.
• Your Crusoe project ID. Find it with crusoe projects list.
• OODLE_INSTANCE: Your Oodle instance ID. Go to the Settings icon -> API Keys page in your Oodle UI to find it. (Oodle UI links: ap1, us1)
• OODLE_API_KEY: Your Oodle API key for authentication. Go to the Settings icon -> API Keys in your Oodle UI to choose an appropriate key. Use a dedicated key, so that you can revoke it without effect on other senders. (Oodle UI links: ap1, us1)
• An Oodle account (navigate to ap1, us1 to start setup). The Crusoe tile shows every snippet below with your instance values filled in.

OpenTelemetry streaming

Crusoe runs a managed forwarder for each forwarding configuration. The forwarder sends metrics and logs to Oodle over OTLP/HTTP as they arrive, CMK control plane audit logs included.

You also need:

• An organization with OpenTelemetry streaming turned on. Contact your Crusoe account team or Crusoe support.
• The Editor role on the Crusoe project.
• A Crusoe API access key to sign the requests. See the Crusoe API reference.
• jq, to build the request body.

1. Choose the data to forward

Save this as filters.yaml. It forwards all metrics and all logs.

metrics: {}
logs: {}

To forward less, add OpenTelemetry Collector filter or transform processors under each signal. For example, this file forwards all metrics and only the CMK control plane audit logs:

metrics: {}
logs:
  filter/audit_only:
    error_mode: ignore
    logs:
      log_record:
        - attributes["log_source"] != "k8s_audit"
  processing_sequence:
    - filter/audit_only

See Filter and transform telemetry for all the rules.

2. Build the request body

The Crusoe forwarder sends one authentication header only. Oodle reads the instance ID and the API key together from the X-OODLE-INSTANCE-API-KEY header, as <OODLE_INSTANCE>/<OODLE_API_KEY>.

jq -n \
  --rawfile filters filters.yaml \
  --arg token "<OODLE_INSTANCE>/<OODLE_API_KEY>" \
  '{
    exporter_name: "oodle",
    destination_type: "otlp_http",
    endpoint_url: "https://<OODLE_INSTANCE>-otlp.collector.oodle.ai",
    auth_type: "custom_header",
    credential: {
      config: { header_name: "X-OODLE-INSTANCE-API-KEY" },
      secret: ({ token: $token } | tojson)
    },
    filters_yaml: $filters
  }' > create.json

Crusoe adds /v1/metrics and /v1/logs to endpoint_url. Crusoe encrypts the credential and never returns it from the API.

3. Create the forwarding configuration

Sign the request as the Crusoe API reference describes, and set $project_id, $timestamp and $signature.

curl -X POST "https://api.crusoecloud.com/v1/projects/$project_id/otel-forwarding-configurations" \
  -H "X-Crusoe-Timestamp: $timestamp" \
  -H "Authorization: Bearer $signature" \
  -H "Content-Type: application/json" \
  -d @create.json

The response contains the exporter_id of the configuration.

4. Wait for the forwarder

Describe the configuration until status.state is running:

curl "https://api.crusoecloud.com/v1/projects/$project_id/otel-forwarding-configurations/$exporter_id" \
  -H "X-Crusoe-Timestamp: $timestamp" \
  -H "Authorization: Bearer $signature"

If status.state is failed, status.message gives the reason. To stop, change or delete the configuration, see Manage forwarding configurations.

Delivery

• The forwarder retries deliveries that fail because of a network error. If Oodle cannot be reached for a long time, the data of that period is not sent later.
• Metrics use cumulative temporality.

Scrape endpoint

Telemetry Conduit serves the metrics of a project at a Prometheus-compatible scrape endpoint. An OpenTelemetry Collector scrapes it and sends the metrics to Oodle. This method sends metrics only.

1. Create a Crusoe monitoring token

crusoe monitoring tokens create

Crusoe shows the token only once, so keep it in a safe place.

2. Configure the collector

Save this as otel-collector-config.yaml. The collector reads the Crusoe project ID, the monitoring token and the Oodle API key from the environment, so the file holds no secrets.

receivers:
  prometheus/crusoe:
    config:
      scrape_configs:
        - job_name: "crusoe"
          # Crusoe collects metrics every 60 seconds and does not accept
          # a shorter scrape interval.
          scrape_interval: 60s
          scrape_timeout: 30s
          # Keep the labels that Crusoe puts on each series, such as
          # instance. Without this, the scrape target overwrites them.
          honor_labels: true
          scheme: https
          metrics_path: "/v1/projects/${env:CRUSOE_PROJECT_ID}/metrics/scrape"
          authorization:
            type: Bearer
            credentials: "${env:CRUSOE_MONITORING_TOKEN}"
          static_configs:
            - targets: ["api.crusoecloud.com"]

processors:
  # Crusoe's reference collector config drops the crusoe_resource label.
  # Remove this processor to keep it.
  transform/crusoe:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "crusoe_resource")
  batch:
    timeout: 10s

exporters:
  otlphttp/oodle:
    endpoint: "https://<OODLE_INSTANCE>-otlp.collector.oodle.ai"
    headers:
      X-OODLE-INSTANCE: "<OODLE_INSTANCE>"
      X-API-KEY: "${env:OODLE_API_KEY}"

service:
  pipelines:
    metrics:
      receivers: [prometheus/crusoe]
      processors: [transform/crusoe, batch]
      exporters: [otlphttp/oodle]

To send only some metrics, add Crusoe's filters to the scrape job as params. Prometheus adds them to the scrape URL as query parameters:

          params:
            metric_name: ["crusoe_vm_memory_.*"]
            metric_category: ["system"]

Parameter	Filters by	Example
metric_name	Metric name, as a comma-separated list	crusoe_vm_memory_.*
labels	Label key:value pairs, as a comma-separated list	collector:disk,device:vda
metric_category	Category: system or custom	system

3. Run the collector

Use the contrib distribution of the collector: the core distribution has no prometheus receiver.

docker run -d --name otelcol-crusoe \
  -e CRUSOE_PROJECT_ID="<crusoe-project-id>" \
  -e CRUSOE_MONITORING_TOKEN="<crusoe-monitoring-token>" \
  -e OODLE_API_KEY="<OODLE_API_KEY>" \
  -v "$PWD/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.164.0

The collector can run anywhere with access to the internet, for example on a Crusoe VM or as a Deployment in a CMK cluster.

Verify

1. Go to ap1, us1 and search for crusoe_vm_ or, on GPU VMs and nodes, DCGM_FI_DEV_GPU_UTIL. Data arrives within a few minutes.
2. With OpenTelemetry streaming, go to ap1, us1 and filter on the log_source attribute, for example journald or k8s_audit.

Data collected

Metrics

Metrics keep the names that Crusoe gives them. For example:

Metrics	Covers
DCGM_FI_*	NVIDIA GPU utilization, memory, temperature, power and errors, from DCGM
crusoe_vm_*	CPU, memory, disk and network of each VM or node
InfiniBand and NVLink metrics	Interconnect throughput and errors
Custom metrics	Metrics that your CMK workloads expose for the Crusoe Watch Agent to scrape

See Crusoe metrics for the complete list.

Logs

OpenTelemetry streaming sends logs. The scrape endpoint does not.

Family	log_source values
Node logs	journald, crusoe-watch-agent, cwa-manager, cwa-report-runner
CMK v2 control plane logs	k8s_audit, k8s_apiserver, k8s_controller_manager, k8s_etcd, k8s_konnectivity, k8s_scheduler

The resource attributes crusoe_org_id and crusoe_project_id are on every record. Node logs also have crusoe_vm_id, crusoe_vm_name and crusoe_cluster_id. Control plane logs carry crusoe_cluster_id as a record attribute.

Each record holds the original line in payload.message and a parsed copy in payload.parsed. To make audit logs about half the size, drop payload.message with a transform processor in filters.yaml.

For most journald logs, the record time is when Crusoe received the line, a few seconds after the event. The event time is in the payload.timestamp attribute.

Troubleshooting

Issue	Resolution
No data arrives	Make sure that the Crusoe Watch Agent runs on your VMs or CMK nodes.
The Crusoe API returns 401 or 403 when you create a configuration	Make sure that you have the Editor role, that the request signature is correct, and that Crusoe turned on OpenTelemetry streaming for your organization.
status.state is failed	Read status.message. A processor in filters.yaml that the forwarder cannot load causes this state.
The forwarder runs, but no data arrives in Oodle	Make sure that the header name is X-OODLE-INSTANCE-API-KEY and that the token is <OODLE_INSTANCE>/<OODLE_API_KEY>. To change the credential, send a PATCH with the complete credential.
The collector logs scrape errors	Run curl -H "Authorization: Bearer <crusoe-monitoring-token>" https://api.crusoecloud.com/v1/projects/<crusoe-project-id>/metrics/scrape and make sure that it returns metrics.
The collector logs export errors	Make sure that OODLE_API_KEY and the instance ID in the exporter are correct.