Skip to main content

Convox ECS

Send metrics, logs, and traces from your Convox (generation 2, ECS/EC2) apps to Oodle, with no application code changes:

  • Logs ride a host-local Fluent Bit agent over syslog, which resolves the app identity per record (no per-app configuration). During migration it can dual-write to CloudWatch too, so nothing downstream breaks while you validate Oodle. The logs setup is covered first, below.
  • Metrics and traces ride the Datadog Agent running in your rack, dual-writing to Oodle alongside Datadog or shipping to Oodle only. See Metrics and traces via the Datadog Agent.

How it works​

Convox's LogDriver=Syslog runs Docker's syslog driver with no tag, so the rfc5424 APP-NAME field carries only the container short-id (e.g. 02a2b9e2f2c0), not the app name, and there is no Convox param to set the tag. A single shared agent therefore can't tell apps apart from the payload alone.

Fluent Bit's ecs filter resolves that short-id to ECS task metadata (the app name) by querying the ECS Agent introspection API. On Convox, agents run in ECS bridge networking, so the introspection endpoint is reachable on the docker bridge gateway 172.17.0.1:51678, not on 127.0.0.1.

app (LogDriver=Syslog, dest tcp://localhost:5140)
   │  Docker syslog rfc5424: APP-NAME = container short-id
   ▼
Fluent Bit agent (one per host, host port 5140)
   syslog input ──▶ rewrite_tag (short-id → tag) ──▶ ecs filter (introspection @172.17.0.1:51678)
                 ──▶ lua (clean app name → Oodle canonical `service`)
   ├─ cloudwatch_logs ──▶ <rack>-<app>-LogGroup-oodle  (dual-write only, one group per app)
   └─ http            ──▶ Oodle /ingest/v1/logs    (JSON, gzip)

Oodle canonical field mapping​

The agent emits Oodle's indexed top-level log field names so enrichment is directly filterable in Oodle instead of buried in the nested log.* blob:

Source (ECS introspection / syslog)Oodle canonical field
clean app name (from $TaskDefinitionFamily)service
$TaskDefinitionFamilytask_definition
$TaskIDtask_id
$ECSContainerNamecontainer_name
container short-id (syslog appname)container_id
$ClusterNamecluster

After this, service = <app> is a first-class filter in the Oodle Logs Explorer and CLI.

Requirements

Convox generation 2 on the EC2 launch type. The ECS introspection API is not available on Fargate.

Configuration​

You'll need the following values:

  • OODLE_INSTANCE: Your Oodle instance ID. Go to the Settings icon → API Keys page in your Oodle UI. (Oodle UI links: ap1, us1)
  • OODLE_API_KEY: Your Oodle API key for authentication. Go to the Settings icon → API Keys in your Oodle UI to choose an appropriate key. (Oodle UI links: ap1, us1)
  • RACK_PREFIX: The short rack name that prefixes every ECS task family (e.g. gm-test), used to derive clean app names.

The agent is a small Convox app made of five files. Create a directory (e.g. oodle-log-agent/) containing the files below. The only file that differs between the two modes is fluent-bit.conf. Pick the tab that matches your phase.

Dual-write ships every log record to both CloudWatch and Oodle, so nothing downstream breaks while you validate Oodle. This fluent-bit.conf has both the cloudwatch_logs and Oodle http outputs:

fluent-bit.conf
# Fluent Bit host-local log agent for Convox (generation 2, ECS/EC2).
#
# Pipeline:
#   syslog(5140) -> rewrite_tag(container short-id -> tag) -> ecs(introspection lookup)
#                -> lua(clean app name) -> { CloudWatch per-app group, Oodle HTTP }
#
# DUAL-WRITE phase (CloudWatch + Oodle). To move to SINGLE-WRITE (Oodle only), delete the
# `cloudwatch_logs` OUTPUT block below (and drop the agent's CloudWatch IAM) and redeploy.

[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  /fluent-bit/etc/parsers-convox.conf
    Daemon        off

[INPUT]
    Name          syslog
    Mode          tcp
    Listen        0.0.0.0
    Port          5140
    Parser        syslog-rfc5424-convox
    Tag           raw.syslog

# Move the container short-id (appname) into the tag; the ecs filter keys off the tag.
[FILTER]
    Name          rewrite_tag
    Match         raw.syslog
    Rule          $appname ^([0-9a-f]{12,64})$ ecs.$appname false
    Emitter_Name  re_ecs

# Enrich with ECS task metadata via the host-local introspection API (bridge gateway).
[FILTER]
    Name             ecs
    Match            ecs.*
    ECS_Meta_Host    172.17.0.1
    ECS_Meta_Port    51678
    ECS_Tag_Prefix   ecs.
    Add              task_definition $TaskDefinitionFamily
    Add              task_id $TaskID
    Add              container_name $ECSContainerName
    Add              cluster $ClusterName

# Reduce the ECS family to a clean Convox app name and map to Oodle's canonical `service`.
[FILTER]
    Name          lua
    Match         ecs.*
    Script        /fluent-bit/etc/derive_app.lua
    Call          derive_app

# ======================================================================================
# DUAL-WRITE ONLY — CloudWatch, mirroring Convox's native names as closely as possible:
#   group  = <rack>-<app>-LogGroup-oodle    (Convox: <rack>-<app>-LogGroup-<cfn-hash>; `cw_group`
#            is composed in derive_app.lua because templates can't put a hyphen after a $variable)
#   stream = service.<container>.<task-id>  (Convox: service/<container>/<task-id>; the core
#            cloudwatch_logs plugin only allows `.`/`,` between template variables, not `/`)
# container_name and task_id come from the `ecs` filter above. Auto-created groups get a 7-day
# retention (Log_Retention_Days), so agent-owned groups self-expire instead of growing forever.
# Log_Key ships ONLY the raw `message` value to CloudWatch (matching Convox's native plain-text
# streams) instead of the full enriched JSON record; the Oodle HTTP output below still gets the
# structured metadata. Delete this whole OUTPUT block (and the agent's CloudWatch IAM) to switch
# to SINGLE-WRITE. Records that failed enrichment fall back to unresolved-LogGroup-oodle, never dropped.
# ======================================================================================
[OUTPUT]
    Name                cloudwatch_logs
    Match               ecs.*
    Region              us-east-1
    Log_Group_Name      unresolved-LogGroup-oodle
    Log_Group_Template  $cw_group
    Log_Stream_Name     service.unresolved
    Log_Stream_Template service.$container_name.$task_id
    Auto_Create_Group   On
    Log_Retention_Days  7
    Log_Key             message

# --- Oodle: native HTTP JSON logs ingest. Path is /ingest/v1/logs (NOT the OTLP path).
#     Host/key come from the environment (never committed). Stays after CloudWatch is removed.
[OUTPUT]
    Name             http
    Match            ecs.*
    Host             ${OODLE_INSTANCE}-logs.collector.oodle.ai
    Port             443
    Uri              /ingest/v1/logs
    Format           json
    Compress         gzip
    Tls              On
    Header           X-OODLE-INSTANCE ${OODLE_INSTANCE}
    Header           X-API-KEY ${OODLE_API_KEY}
    Json_date_key    timestamp
    Json_date_format iso8601

# --- Debug (optional): uncomment to echo enriched records to this agent's own log while
#     validating. Keep it off in production.
# [OUTPUT]
#     Name          stdout
#     Match         ecs.*
#     Format        json_lines

The remaining four files are identical for both modes:

parsers-convox.conf
[PARSER]
    Name        syslog-rfc5424-convox
    Format      regex
    Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<hostname>[^ ]+) (?<appname>[^ ]+) (?<procid>[-0-9]+) (?<msgid>[^ ]+) (?<sd>(\[.*\]|-)) (?<message>.+)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On
derive_app.lua
-- Reduce the ECS task-definition family (added by the `ecs` filter as `task_definition`) to a
-- clean Convox app name, and map it onto Oodle's canonical `service` field. Also promote the
-- container short-id into Oodle's canonical `container_id`.
--
-- Family shape:
--   <rack>-<app>-Service<Kind>-<cfnRandom>-service-<svc>
--   e.g. gm-test-rails-demo-ServiceWeb-CRFBDIVNUAUO-service-web  ->  service = rails-demo
local RACK_PREFIX = os.getenv("RACK_PREFIX") or ""

local function escape(s)
    return (s:gsub("([%-%.%+%[%]%(%)%$%^%%%?%*])", "%%%1"))
end

function derive_app(tag, ts, record)
    local fam = record["task_definition"]
    if fam == nil then
        return 0, ts, record   -- 0 = leave record unchanged (no metadata to work with)
    end
    local app = fam
    local base = string.match(fam, "^(.-)%-Service")   -- strip "-Service<Kind>-..." suffix
    if base ~= nil then app = base end
    if RACK_PREFIX ~= "" then
        local stripped = string.match(app, "^" .. escape(RACK_PREFIX) .. "%-(.+)$")
        if stripped ~= nil then app = stripped end     -- strip "<rack>-" prefix
    end
    record["service"] = app                            -- Oodle canonical app field
    -- CloudWatch group that mirrors Convox's native <rack>-<app>-LogGroup-<hash>, with a fixed
    -- "oodle" suffix in place of the CloudFormation hash. Composed here rather than in the output
    -- template because cloudwatch_logs templates can't place a hyphen after a $variable.
    if RACK_PREFIX ~= "" then
        record["cw_group"] = RACK_PREFIX .. "-" .. app .. "-LogGroup-oodle"
    else
        record["cw_group"] = app .. "-LogGroup-oodle"
    end
    if record["appname"] ~= nil then
        record["container_id"] = record["appname"]     -- Oodle canonical container id (short)
    end
    return 2, ts, record       -- 2 = record modified
end
convox.yml
environment:
  - OODLE_INSTANCE
  - OODLE_API_KEY
  - RACK_PREFIX
services:
  collector:
    agent:
      enabled: true
      ports:
        - 5140/tcp
    build: .
    scale:
      cpu: 128
      memory: 256
Dockerfile
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable

COPY fluent-bit.conf     /fluent-bit/etc/fluent-bit.conf
COPY parsers-convox.conf /fluent-bit/etc/parsers-convox.conf
COPY derive_app.lua      /fluent-bit/etc/derive_app.lua

Deploy​

1. Protect existing CloudWatch history (do this first)​

Switching an app off Convox's native CloudWatch driver deletes its Convox-managed log group and all history. Because step 5 switches every app in the rack over to the agent, protect every app first. Each app's log group is owned by its own CloudFormation stack (<rack>-<app>) as a resource with the logical id LogGroup (the physical CloudWatch name is auto-generated, e.g. gm-test-rails-demo-LogGroup-qusbLg1J0t2a). Setting DeletionPolicy: Retain on that resource before switching makes the switch orphan the group — it stays in place with the same name and all history — instead of deleting it.

All commands here need jq. First, audit what's already set — this is read-only and tells you whether any app still needs protecting (YES = its group would be deleted on the switch):

set -euo pipefail
RACK=<rack>
REGION=us-east-1

printf '%-28s %-16s %s\n' "APP STACK" "DELETIONPOLICY" "NEEDS UPDATE?"
aws cloudformation describe-stacks --region "$REGION" \
  --query "Stacks[?Tags[?Key=='System'&&Value=='convox']]|[?Tags[?Key=='Type'&&Value=='app']]|[?Tags[?Key=='Rack'&&Value=='${RACK}']].StackName" \
  --output text | tr '\t' '\n' | while read -r STACK; do
  dp=$(aws cloudformation get-template --region "$REGION" --stack-name "$STACK" \
        --query TemplateBody --output json \
        | jq -r '.Resources.LogGroup.DeletionPolicy // (if .Resources.LogGroup then "none" else "no-loggroup" end)')
  case "$dp" in
    Retain)      need="no  (protected)";;
    no-loggroup) need="no  (nothing to protect)";;
    *)           need="YES (would be deleted on switch)";;
  esac
  printf '%-28s %-16s %s\n' "$STACK" "$dp" "$need"
done
Example output
APP STACK                    DELETIONPOLICY   NEEDS UPDATE?
gm-test-datadog-agent        none             YES (would be deleted on switch)
gm-test-rails-demo           none             YES (would be deleted on switch)
gm-test-gm-test-app          Retain           no  (protected)

If every app already reads Retain, you're done — skip the rest of this step. Otherwise run the loop below. It finds every Convox app stack in the rack (by tags), and for each one adds DeletionPolicy: Retain to its LogGroup, skipping apps that have no log group or are already protected (so it is safe to re-run):

set -euo pipefail
RACK=<rack>
REGION=us-east-1

# Rack settings bucket, used to stage each template. CloudFormation's inline --template-body
# limit is 51,200 bytes, so we always upload to S3 and update via --template-url, which works
# for any template size.
BUCKET=$(aws s3api list-buckets \
  --query "Buckets[?starts_with(Name,'${RACK}-settings-')].Name | [0]" --output text)

# Every Convox app stack in the rack (tagged System=convox, Type=app, Rack=<rack>).
APPS=$(aws cloudformation describe-stacks --region "$REGION" \
  --query "Stacks[?Tags[?Key=='System'&&Value=='convox']]|[?Tags[?Key=='Type'&&Value=='app']]|[?Tags[?Key=='Rack'&&Value=='${RACK}']].StackName" \
  --output text)

for STACK in $APPS; do
  aws cloudformation get-template --region "$REGION" --stack-name "$STACK" \
    --query TemplateBody --output json > template.json

  # Skip apps with no Convox-managed log group, or already protected.
  if [ "$(jq -r '.Resources.LogGroup // "MISSING"' template.json)" = MISSING ]; then
    echo "$STACK: no LogGroup, skipping"; continue
  fi
  if [ "$(jq -r '.Resources.LogGroup.DeletionPolicy // "none"' template.json)" = Retain ]; then
    echo "$STACK: already Retain, skipping"; continue
  fi

  # Add DeletionPolicy: Retain to the LogGroup resource.
  jq '.Resources.LogGroup.DeletionPolicy = "Retain"' template.json > retain.json

  # A full-template update must resubmit ALL parameters; reuse each existing value.
  aws cloudformation describe-stacks --region "$REGION" --stack-name "$STACK" \
    --query "Stacks[0].Parameters[].ParameterKey" --output json \
    | jq '[.[] | {ParameterKey: ., UsePreviousValue: true}]' > params.json

  # Stage the template in S3 (sidesteps the 51,200-byte inline limit), then update via URL.
  aws s3 cp retain.json "s3://${BUCKET}/tmp/${STACK}-retain.json" --region "$REGION"

  aws cloudformation update-stack --region "$REGION" --stack-name "$STACK" \
    --template-url "https://s3.${REGION}.amazonaws.com/${BUCKET}/tmp/${STACK}-retain.json" \
    --parameters file://params.json \
    --capabilities CAPABILITY_IAM
  echo "$STACK: Retain applied"
done

Wait for each update to finish (aws cloudformation wait stack-update-complete --region "$REGION" --stack-name <rack>-<app>). The update touches only the LogGroup (action Modify, no replacement) plus each app's service/resource sub-stacks as no-ops. After this, every original group is kept in place with all its history, and the agent writes new logs to <rack>-<app>-LogGroup-oodle.

Protecting a single app instead

To retain just one app (e.g. you are opting in per app rather than flipping the whole rack), run the body of the loop for that one <rack>-<app> stack.

The rack's own system logs

The rack stack (<rack>) owns its own LogGroup for the rack's system logs. Re-submitting its template re-evaluates ~13 rack resources (autoscaling groups, launch templates, the API ECS service), the same class of change as convox rack update, which can churn instances. Only protect the rack's system-log group if you need that history, and do it in a maintenance window; the app loop above does not touch it.

2. Create the agent app and set credentials​

convox apps create oodle-log-agent -r <rack>

convox env set \
  OODLE_INSTANCE=<instance-id> \
  OODLE_API_KEY=<api-key> \
  RACK_PREFIX=<rack-prefix> \
  -a oodle-log-agent -r <rack>

3. Grant CloudWatch write access (dual-write only)​

convox apps params set IamPolicy=arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
  -a oodle-log-agent -r <rack>

4. Deploy the agent​

Deploy the agent before pointing any app at it. Its syslog driver needs something listening on the host port or its containers fail to start:

convox deploy -a oodle-log-agent -r <rack>

5. Point logs at the agent​

Set the syslog driver as the rack default. This applies to newly created apps and to any app on its next deploy. It does not retroactively move apps that are already running (handle those below):

convox rack params set LogDriver=Syslog SyslogDestination=tcp://localhost:5140 SyslogFormat=rfc5424 \
  -r <rack>
Exclude the agent app itself

The rack command above also flips the oodle-log-agent app onto the syslog driver, so the agent ships its own container logs back into its own syslog input on port 5140. Fluent Bit's internal lines carry whole-second timestamps that don't match the parser's .%L%z format, producing a repeating invalid time format ... for '2026-07-23T04:04:31Z' warning that also lands in Oodle under service=oodle-log-agent. Keep the agent on its native CloudWatch driver so it never feeds itself:

convox apps params set LogDriver=CloudWatch SyslogDestination="" -a oodle-log-agent -r <rack>

Switch your existing apps. The rack default does not move apps that are already running, so switch each one explicitly (skip oodle-log-agent). To move every existing app at once:

for app in $(convox apps -r <rack> | tail -n +2 | awk '{print $1}' | grep -vx oodle-log-agent); do
  convox apps params set LogDriver=Syslog SyslogDestination=tcp://localhost:5140 SyslogFormat=rfc5424 -a "$app" -r <rack>
done

6. Verify both paths​

Confirm both destinations: the CloudWatch group <rack>-<app>-LogGroup-oodle is auto-created, and in Oodle the logs are filterable by service=<app>. Allow a few minutes for data to start flowing.

To eyeball enrichment at the record level, uncomment the stdout output in fluent-bit.conf and tail the agent:

convox logs -a oodle-log-agent -r <rack>   # shows enriched records once stdout is enabled

Reverting​

To move a single app back to Convox's native CloudWatch driver (this restores convox logs -a <app>):

convox apps params set LogDriver=CloudWatch SyslogDestination="" -a <app> -r <rack>

If you switched the whole rack, revert it the same way at the rack level:

convox rack params set LogDriver=CloudWatch SyslogDestination="" -r <rack>

Metrics and traces via the Datadog Agent​

If you already run the Datadog Agent as a Convox app in your rack, you can ship its metrics and APM traces to Oodle. This is agent-side only (no application code changes). Pick the mode that matches your phase:

  • Dual-write: the Agent keeps shipping to Datadog and forwards a copy to Oodle, so nothing downstream breaks while you validate Oodle.
  • Single-write: Oodle is the only intake and nothing is sent to Datadog. No Datadog account is required.

Both use the Datadog intake paths on the metrics collector domain. Because the values carry your Oodle API key, the manifest declares only the (bare) names and the values are injected as secrets with convox env set.

The Agent ships primarily to Datadog and forwards a copy to Oodle via the additional-endpoints vars, each a JSON map of {endpoint_url: [api_key]}. Your Datadog DD_API_KEY / DD_SITE stay unchanged; the Oodle key travels inside the JSON.

SignalEnv varOodle endpoint
MetricsDD_ADDITIONAL_ENDPOINTShttps://<instance-id>.collector.oodle.ai/v1/datadog/<instance-id>
TracesDD_APM_ADDITIONAL_ENDPOINTShttps://<instance-id>.collector.oodle.ai/v1/datadog_traces/<instance-id>

1. Declare the endpoint vars in the Agent's manifest​

Convox only injects env vars that a service's manifest declares, so add both names to the environment list in the Datadog Agent app's convox.yml before setting their values. They are bare (no values here). DD_API_KEY / DD_SITE are your existing Datadog credentials:

convox.yml
environment:
  - DD_API_KEY                    # your Datadog key (unchanged)
  - DD_SITE                       # your Datadog site (unchanged)
  - DD_ADDITIONAL_ENDPOINTS       # metrics copy -> Oodle
  - DD_USE_V3_API_SERIES_ENABLED  # metrics: use v2 series intake Oodle accepts
  - DD_APM_ADDITIONAL_ENDPOINTS   # traces  copy -> Oodle

2. Inject the Oodle endpoint values as secrets​

This is additive, so your existing DD_API_KEY / DD_SITE are untouched. Single-quote each value so the shell leaves the JSON braces intact:

convox env set \
  'DD_ADDITIONAL_ENDPOINTS={"https://<instance-id>.collector.oodle.ai/v1/datadog/<instance-id>":["<api-key>"]}' \
  DD_USE_V3_API_SERIES_ENABLED=false \
  'DD_APM_ADDITIONAL_ENDPOINTS={"https://<instance-id>.collector.oodle.ai/v1/datadog_traces/<instance-id>":["<api-key>"]}' \
  -a datadog-agent -r <rack>

3. Deploy the Agent​

convox env set alone does not roll a running daemon. Deploy (or promote a release) so the new values reach the container:

convox deploy -a datadog-agent -r <rack>

Once traffic flows, Datadog-origin metrics (for example dd_trace_stats_hits and other datadog_* series) and your APM spans appear in Oodle, filterable by service. Allow a few minutes for data to start flowing.

Logs too

These vars cover metrics and traces. To also ship logs from the Agent, enable logs and add DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS (dual-write) or DD_LOGS_CONFIG_LOGS_DD_URL (single-write). The Datadog integration guide covers all signals and every configuration method.

Notes and limitations​

  • EC2 launch type only. The ecs filter's introspection API is not available on Fargate.
  • 172.17.0.1 is the Docker default bridge gateway (stable on the ECS-optimized AMI). If a rack uses a non-default bridge, derive the gateway at start instead of hardcoding.
  • convox scale collector --count 0 does not stop the agent (it's a DaemonSet). Delete the app to actually stop it.
  • Non-rfc5424 lines (occasional Convox framing) fail the parser and are dropped. This is harmless.
  • The debug stdout output is commented out by default; uncomment it while validating to echo enriched records to the agent's own log, and keep it off in production.

Support

If you need assistance or have any questions, please reach out to us through: