Skip to main content

Convox ECS

Send metrics, logs, and traces from your Convox (generation 2, ECS/EC2) apps to Oodle, with no application code changes:

  • Logs ride a host-local Fluent Bit agent over syslog, which resolves the app identity per record (no per-app configuration). During migration it can dual-write to CloudWatch too, so nothing downstream breaks while you validate Oodle. The logs setup is covered first, below.
  • Metrics and traces ride the Datadog Agent running in your rack, dual-writing to Oodle alongside Datadog or shipping to Oodle only. See Metrics and traces via the Datadog Agent.

How it works​

Convox's LogDriver=Syslog runs Docker's syslog driver with no tag, so the rfc5424 APP-NAME field carries only the container short-id (e.g. 02a2b9e2f2c0), not the app name, and there is no Convox param to set the tag. A single shared agent therefore can't tell apps apart from the payload alone.

Fluent Bit's ecs filter resolves that short-id to ECS task metadata (the app name) by querying the ECS Agent introspection API. On Convox, agents run in ECS bridge networking, so the introspection endpoint is reachable on the docker bridge gateway 172.17.0.1:51678, not on 127.0.0.1.

app (LogDriver=Syslog, dest tcp://localhost:5140)
   │  Docker syslog rfc5424: APP-NAME = container short-id
   ▼
Fluent Bit agent (one per host, host port 5140)
   syslog input ──▶ rewrite_tag (short-id → tag) ──▶ ecs filter (introspection @172.17.0.1:51678)
                 ──▶ lua (clean app name → Oodle canonical `service`)
   ├─ cloudwatch_logs ──▶ /convox/<app>            (dual-write only, one group per app)
   └─ http            ──▶ Oodle /ingest/v1/logs    (JSON, gzip)

Oodle canonical field mapping​

The agent emits Oodle's indexed top-level log field names so enrichment is directly filterable in Oodle instead of buried in the nested log.* blob:

Source (ECS introspection / syslog)Oodle canonical field
clean app name (from $TaskDefinitionFamily)service
$TaskDefinitionFamilytask_definition
$TaskIDtask_id
$ECSContainerNamecontainer_name
container short-id (syslog appname)container_id
$ClusterNamecluster

After this, service = <app> is a first-class filter in the Oodle Logs Explorer and CLI.

Requirements

Convox generation 2 on the EC2 launch type. The ECS introspection API is not available on Fargate.

Configuration​

You'll need the following values:

  • OODLE_INSTANCE: Your Oodle instance ID. Go to the Settings icon → API Keys page in your Oodle UI. (Oodle UI links: ap1, us1)
  • OODLE_API_KEY: Your Oodle API key for authentication. Go to the Settings icon → API Keys in your Oodle UI to choose an appropriate key. (Oodle UI links: ap1, us1)
  • RACK_PREFIX: The short rack name that prefixes every ECS task family (e.g. gm-test), used to derive clean app names.

The agent is a small Convox app made of five files. Create a directory (e.g. oodle-log-agent/) containing the files below. The only file that differs between the two modes is fluent-bit.conf. Pick the tab that matches your phase.

Dual-write ships every log record to both CloudWatch and Oodle, so nothing downstream breaks while you validate Oodle. This fluent-bit.conf has both the cloudwatch_logs and Oodle http outputs:

fluent-bit.conf
# Fluent Bit host-local log agent for Convox (generation 2, ECS/EC2).
#
# Pipeline:
#   syslog(5140) -> rewrite_tag(container short-id -> tag) -> ecs(introspection lookup)
#                -> lua(clean app name) -> { CloudWatch per-app group, Oodle HTTP }
#
# DUAL-WRITE phase (CloudWatch + Oodle). To move to SINGLE-WRITE (Oodle only), delete the
# `cloudwatch_logs` OUTPUT block below (and drop the agent's CloudWatch IAM) and redeploy.

[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  /fluent-bit/etc/parsers-convox.conf
    Daemon        off

[INPUT]
    Name          syslog
    Mode          tcp
    Listen        0.0.0.0
    Port          5140
    Parser        syslog-rfc5424-convox
    Tag           raw.syslog

# Move the container short-id (appname) into the tag; the ecs filter keys off the tag.
[FILTER]
    Name          rewrite_tag
    Match         raw.syslog
    Rule          $appname ^([0-9a-f]{12,64})$ ecs.$appname false
    Emitter_Name  re_ecs

# Enrich with ECS task metadata via the host-local introspection API (bridge gateway).
[FILTER]
    Name             ecs
    Match            ecs.*
    ECS_Meta_Host    172.17.0.1
    ECS_Meta_Port    51678
    ECS_Tag_Prefix   ecs.
    Add              task_definition $TaskDefinitionFamily
    Add              task_id $TaskID
    Add              container_name $ECSContainerName
    Add              cluster $ClusterName

# Reduce the ECS family to a clean Convox app name and map to Oodle's canonical `service`.
[FILTER]
    Name          lua
    Match         ecs.*
    Script        /fluent-bit/etc/derive_app.lua
    Call          derive_app

# ======================================================================================
# DUAL-WRITE ONLY — CloudWatch, one log group PER APP, templated from the derived name.
# Delete this whole OUTPUT block (and the agent's CloudWatch IAM) to switch to SINGLE-WRITE.
# Records that failed enrichment fall back to /convox/unresolved, never dropped.
# ======================================================================================
[OUTPUT]
    Name                cloudwatch_logs
    Match               ecs.*
    Region              us-east-1
    Log_Group_Name      /convox/unresolved
    Log_Group_Template  /convox/$service
    Log_Stream_Name     web
    Auto_Create_Group   On

# --- Oodle: native HTTP JSON logs ingest. Path is /ingest/v1/logs (NOT the OTLP path).
#     Host/key come from the environment (never committed). Stays after CloudWatch is removed.
[OUTPUT]
    Name             http
    Match            ecs.*
    Host             ${OODLE_INSTANCE}-logs.collector.oodle.ai
    Port             443
    Uri              /ingest/v1/logs
    Format           json
    Compress         gzip
    Tls              On
    Header           X-OODLE-INSTANCE ${OODLE_INSTANCE}
    Header           X-API-KEY ${OODLE_API_KEY}
    Json_date_key    timestamp
    Json_date_format iso8601

# --- Debug (optional): uncomment to echo enriched records to this agent's own log while
#     validating. Keep it off in production.
# [OUTPUT]
#     Name          stdout
#     Match         ecs.*
#     Format        json_lines

The remaining four files are identical for both modes:

parsers-convox.conf
[PARSER]
    Name        syslog-rfc5424-convox
    Format      regex
    Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<hostname>[^ ]+) (?<appname>[^ ]+) (?<procid>[-0-9]+) (?<msgid>[^ ]+) (?<sd>(\[.*\]|-)) (?<message>.+)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On
derive_app.lua
-- Reduce the ECS task-definition family (added by the `ecs` filter as `task_definition`) to a
-- clean Convox app name, and map it onto Oodle's canonical `service` field. Also promote the
-- container short-id into Oodle's canonical `container_id`.
--
-- Family shape:
--   <rack>-<app>-Service<Kind>-<cfnRandom>-service-<svc>
--   e.g. gm-test-rails-demo-ServiceWeb-CRFBDIVNUAUO-service-web  ->  service = rails-demo
local RACK_PREFIX = os.getenv("RACK_PREFIX") or ""

local function escape(s)
    return (s:gsub("([%-%.%+%[%]%(%)%$%^%%%?%*])", "%%%1"))
end

function derive_app(tag, ts, record)
    local fam = record["task_definition"]
    if fam == nil then
        return 0, ts, record   -- 0 = leave record unchanged (no metadata to work with)
    end
    local app = fam
    local base = string.match(fam, "^(.-)%-Service")   -- strip "-Service<Kind>-..." suffix
    if base ~= nil then app = base end
    if RACK_PREFIX ~= "" then
        local stripped = string.match(app, "^" .. escape(RACK_PREFIX) .. "%-(.+)$")
        if stripped ~= nil then app = stripped end     -- strip "<rack>-" prefix
    end
    record["service"] = app                            -- Oodle canonical app field
    if record["appname"] ~= nil then
        record["container_id"] = record["appname"]     -- Oodle canonical container id (short)
    end
    return 2, ts, record       -- 2 = record modified
end
convox.yml
environment:
  - OODLE_INSTANCE
  - OODLE_API_KEY
  - RACK_PREFIX
services:
  collector:
    agent:
      enabled: true
      ports:
        - 5140/tcp
    build: .
    scale:
      cpu: 128
      memory: 256
Dockerfile
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable

COPY fluent-bit.conf     /fluent-bit/etc/fluent-bit.conf
COPY parsers-convox.conf /fluent-bit/etc/parsers-convox.conf
COPY derive_app.lua      /fluent-bit/etc/derive_app.lua

Deploy​

1. Protect existing CloudWatch history (do this first)​

Switching an app off Convox's native CloudWatch driver deletes its Convox-managed log group and all history. If you want to keep that history, set DeletionPolicy: Retain on the log group before switching, so the switch orphans it (keeps it in place, same name, all history) instead of deleting it. The log group is managed by the app's CloudFormation stack (Convox names it <rack>-<app>, e.g. gm-test-rails-demo):

# 1. Find the app's CloudFormation stack.
aws cloudformation list-stacks \
  --query "StackSummaries[?contains(StackName,'<app>')].StackName"

# 2. Download its template.
aws cloudformation get-template --stack-name <rack>-<app> \
  --region us-east-1 --query TemplateBody --output json > template.json

# 3. Add "DeletionPolicy": "Retain" to the LogGroup resource in template.json
#    (under Resources.LogGroup), then apply the change, reusing existing params:
aws cloudformation update-stack --stack-name <rack>-<app> --region us-east-1 \
  --template-body file://template.json \
  --parameters ParameterKey=<Key>,UsePreviousValue=true \
  --capabilities CAPABILITY_IAM

After this, the original group is kept in place with all its history, and the agent writes new logs to /convox/<app>.

2. Create the agent app and set credentials​

convox apps create oodle-log-agent -r <rack>

convox env set \
  OODLE_INSTANCE=<instance-id> \
  OODLE_API_KEY=<api-key> \
  RACK_PREFIX=<rack-prefix> \
  -a oodle-log-agent -r <rack>

3. Grant CloudWatch write access (dual-write only)​

convox apps params set IamPolicy=arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
  -a oodle-log-agent -r <rack>

4. Deploy the agent​

Deploy the agent before pointing any app at it. Its syslog driver needs something listening on the host port or its containers fail to start:

convox deploy -a oodle-log-agent -r <rack>

5. Point logs at the agent​

Switch the syslog driver at the rack level, so every app (plus the rack's own system logs) routes to the agent in one step, with nothing to repeat per app:

convox rack params set LogDriver=Syslog SyslogDestination=tcp://localhost:5140 SyslogFormat=rfc5424 \
  -r <rack>

Optional (per-app instead): to opt in only specific apps and leave the rest on their native CloudWatch driver, set the same params per app (reversible):

convox apps params set LogDriver=Syslog SyslogDestination=tcp://localhost:5140 SyslogFormat=rfc5424 \
  -a <app> -r <rack>

6. Verify both paths​

Confirm both destinations: the CloudWatch group /convox/<app> is auto-created, and in Oodle the logs are filterable by service=<app>. Allow a few minutes for data to start flowing.

To eyeball enrichment at the record level, uncomment the stdout output in fluent-bit.conf and tail the agent:

convox logs -a oodle-log-agent -r <rack>   # shows enriched records once stdout is enabled

Reverting​

To move a single app back to Convox's native CloudWatch driver (this restores convox logs -a <app>):

convox apps params set LogDriver=CloudWatch SyslogDestination="" -a <app> -r <rack>

If you switched the whole rack, revert it the same way at the rack level:

convox rack params set LogDriver=CloudWatch SyslogDestination="" -r <rack>

Metrics and traces via the Datadog Agent​

If you already run the Datadog Agent as a Convox app in your rack, you can ship its metrics and APM traces to Oodle. This is agent-side only (no application code changes). Pick the mode that matches your phase:

  • Dual-write: the Agent keeps shipping to Datadog and forwards a copy to Oodle, so nothing downstream breaks while you validate Oodle.
  • Single-write: Oodle is the only intake and nothing is sent to Datadog. No Datadog account is required.

Both use the Datadog intake paths on the metrics collector domain. Because the values carry your Oodle API key, the manifest declares only the (bare) names and the values are injected as secrets with convox env set.

The Agent ships primarily to Datadog and forwards a copy to Oodle via the additional-endpoints vars, each a JSON map of {endpoint_url: [api_key]}. Your Datadog DD_API_KEY / DD_SITE stay unchanged; the Oodle key travels inside the JSON.

SignalEnv varOodle endpoint
MetricsDD_ADDITIONAL_ENDPOINTShttps://<instance-id>.collector.oodle.ai/v1/datadog/<instance-id>
TracesDD_APM_ADDITIONAL_ENDPOINTShttps://<instance-id>.collector.oodle.ai/v1/datadog_traces/<instance-id>

1. Declare the endpoint vars in the Agent's manifest​

Convox only injects env vars that a service's manifest declares, so add both names to the environment list in the Datadog Agent app's convox.yml before setting their values. They are bare (no values here). DD_API_KEY / DD_SITE are your existing Datadog credentials:

convox.yml
environment:
  - DD_API_KEY                    # your Datadog key (unchanged)
  - DD_SITE                       # your Datadog site (unchanged)
  - DD_ADDITIONAL_ENDPOINTS       # metrics copy -> Oodle
  - DD_APM_ADDITIONAL_ENDPOINTS   # traces  copy -> Oodle

2. Inject the Oodle endpoint values as secrets​

This is additive, so your existing DD_API_KEY / DD_SITE are untouched. Single-quote each value so the shell leaves the JSON braces intact:

convox env set \
  'DD_ADDITIONAL_ENDPOINTS={"https://<instance-id>.collector.oodle.ai/v1/datadog/<instance-id>":["<api-key>"]}' \
  'DD_APM_ADDITIONAL_ENDPOINTS={"https://<instance-id>.collector.oodle.ai/v1/datadog_traces/<instance-id>":["<api-key>"]}' \
  -a datadog-agent -r <rack>

3. Deploy the Agent​

convox env set alone does not roll a running daemon. Deploy (or promote a release) so the new values reach the container:

convox deploy -a datadog-agent -r <rack>

Once traffic flows, Datadog-origin metrics (for example dd_trace_stats_hits and other datadog_* series) and your APM spans appear in Oodle, filterable by service. Allow a few minutes for data to start flowing.

Logs too

These vars cover metrics and traces. To also ship logs from the Agent, enable logs and add DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS (dual-write) or DD_LOGS_CONFIG_LOGS_DD_URL (single-write). The Datadog integration guide covers all signals and every configuration method.

Notes and limitations​

  • EC2 launch type only. The ecs filter's introspection API is not available on Fargate.
  • 172.17.0.1 is the Docker default bridge gateway (stable on the ECS-optimized AMI). If a rack uses a non-default bridge, derive the gateway at start instead of hardcoding.
  • convox scale collector --count 0 does not stop the agent (it's a DaemonSet). Delete the app to actually stop it.
  • Non-rfc5424 lines (occasional Convox framing) fail the parser and are dropped. This is harmless.
  • The debug stdout output is commented out by default; uncomment it while validating to echo enriched records to the agent's own log, and keep it off in production.

Support

If you need assistance or have any questions, please reach out to us through: