Skip to main content

Convox ECS

Ship logs from your Convox (generation 2, ECS/EC2) apps to Oodle using a host-local Fluent Bit agent. Each app's logs are routed over syslog to the agent, which resolves the app identity per record (with no per-app configuration and no application code changes) and ships them to Oodle. During migration it can dual-write to CloudWatch too, so nothing downstream breaks while you validate Oodle.

How it works​

Convox's LogDriver=Syslog runs Docker's syslog driver with no tag, so the rfc5424 APP-NAME field carries only the container short-id (e.g. 02a2b9e2f2c0), not the app name, and there is no Convox param to set the tag. A single shared agent therefore can't tell apps apart from the payload alone.

Fluent Bit's ecs filter resolves that short-id to ECS task metadata (the app name) by querying the ECS Agent introspection API. On Convox, agents run in ECS bridge networking, so the introspection endpoint is reachable on the docker bridge gateway 172.17.0.1:51678, not on 127.0.0.1.

app (LogDriver=Syslog, dest tcp://localhost:5140)
   │  Docker syslog rfc5424: APP-NAME = container short-id
   ▼
Fluent Bit agent (one per host, host port 5140)
   syslog input ──▶ rewrite_tag (short-id → tag) ──▶ ecs filter (introspection @172.17.0.1:51678)
                 ──▶ lua (clean app name → Oodle canonical `service`)
   ├─ cloudwatch_logs ──▶ /convox/<app>            (dual-write only, one group per app)
   └─ http            ──▶ Oodle /ingest/v1/logs    (JSON, gzip)

Oodle canonical field mapping​

The agent emits Oodle's indexed top-level log field names so enrichment is directly filterable in Oodle instead of buried in the nested log.* blob:

Source (ECS introspection / syslog)Oodle canonical field
clean app name (from $TaskDefinitionFamily)service
$TaskDefinitionFamilytask_definition
$TaskIDtask_id
$ECSContainerNamecontainer_name
container short-id (syslog appname)container_id
$ClusterNamecluster

After this, service = <app> is a first-class filter in the Oodle Logs Explorer and CLI.

Requirements

Convox generation 2 on the EC2 launch type. The ECS introspection API is not available on Fargate.

Configuration​

You'll need the following values:

  • OODLE_INSTANCE: Your Oodle instance ID. Go to the Settings icon → API Keys page in your Oodle UI. (Oodle UI links: ap1, us1)
  • OODLE_API_KEY: Your Oodle API key for authentication. Go to the Settings icon → API Keys in your Oodle UI to choose an appropriate key. (Oodle UI links: ap1, us1)
  • RACK_PREFIX: The short rack name that prefixes every ECS task family (e.g. gm-test), used to derive clean app names.

The agent is a small Convox app made of five files. Create a directory (e.g. oodle-log-agent/) containing the files below. The only file that differs between the two modes is fluent-bit.conf. Pick the tab that matches your phase.

Dual-write ships every log record to both CloudWatch and Oodle, so nothing downstream breaks while you validate Oodle. This fluent-bit.conf has both the cloudwatch_logs and Oodle http outputs:

fluent-bit.conf
# Fluent Bit host-local log agent for Convox (generation 2, ECS/EC2).
#
# Pipeline:
#   syslog(5140) -> rewrite_tag(container short-id -> tag) -> ecs(introspection lookup)
#                -> lua(clean app name) -> { CloudWatch per-app group, Oodle HTTP }
#
# DUAL-WRITE phase (CloudWatch + Oodle). To move to SINGLE-WRITE (Oodle only), delete the
# `cloudwatch_logs` OUTPUT block below (and drop the agent's CloudWatch IAM) and redeploy.

[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  /fluent-bit/etc/parsers-convox.conf
    Daemon        off

[INPUT]
    Name          syslog
    Mode          tcp
    Listen        0.0.0.0
    Port          5140
    Parser        syslog-rfc5424-convox
    Tag           raw.syslog

# Move the container short-id (appname) into the tag; the ecs filter keys off the tag.
[FILTER]
    Name          rewrite_tag
    Match         raw.syslog
    Rule          $appname ^([0-9a-f]{12,64})$ ecs.$appname false
    Emitter_Name  re_ecs

# Enrich with ECS task metadata via the host-local introspection API (bridge gateway).
[FILTER]
    Name             ecs
    Match            ecs.*
    ECS_Meta_Host    172.17.0.1
    ECS_Meta_Port    51678
    ECS_Tag_Prefix   ecs.
    Add              task_definition $TaskDefinitionFamily
    Add              task_id $TaskID
    Add              container_name $ECSContainerName
    Add              cluster $ClusterName

# Reduce the ECS family to a clean Convox app name and map to Oodle's canonical `service`.
[FILTER]
    Name          lua
    Match         ecs.*
    Script        /fluent-bit/etc/derive_app.lua
    Call          derive_app

# ======================================================================================
# DUAL-WRITE ONLY — CloudWatch, one log group PER APP, templated from the derived name.
# Delete this whole OUTPUT block (and the agent's CloudWatch IAM) to switch to SINGLE-WRITE.
# Records that failed enrichment fall back to /convox/unresolved, never dropped.
# ======================================================================================
[OUTPUT]
    Name                cloudwatch_logs
    Match               ecs.*
    Region              us-east-1
    Log_Group_Name      /convox/unresolved
    Log_Group_Template  /convox/$service
    Log_Stream_Name     web
    Auto_Create_Group   On

# --- Oodle: native HTTP JSON logs ingest. Path is /ingest/v1/logs (NOT the OTLP path).
#     Host/key come from the environment (never committed). Stays after CloudWatch is removed.
[OUTPUT]
    Name             http
    Match            ecs.*
    Host             ${OODLE_INSTANCE}-logs.collector.oodle.ai
    Port             443
    Uri              /ingest/v1/logs
    Format           json
    Compress         gzip
    Tls              On
    Header           X-OODLE-INSTANCE ${OODLE_INSTANCE}
    Header           X-API-KEY ${OODLE_API_KEY}
    Json_date_key    timestamp
    Json_date_format iso8601

# --- Debug (optional): uncomment to echo enriched records to this agent's own log while
#     validating. Keep it off in production.
# [OUTPUT]
#     Name          stdout
#     Match         ecs.*
#     Format        json_lines

The remaining four files are identical for both modes:

parsers-convox.conf
[PARSER]
    Name        syslog-rfc5424-convox
    Format      regex
    Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<hostname>[^ ]+) (?<appname>[^ ]+) (?<procid>[-0-9]+) (?<msgid>[^ ]+) (?<sd>(\[.*\]|-)) (?<message>.+)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On
derive_app.lua
-- Reduce the ECS task-definition family (added by the `ecs` filter as `task_definition`) to a
-- clean Convox app name, and map it onto Oodle's canonical `service` field. Also promote the
-- container short-id into Oodle's canonical `container_id`.
--
-- Family shape:
--   <rack>-<app>-Service<Kind>-<cfnRandom>-service-<svc>
--   e.g. gm-test-rails-demo-ServiceWeb-CRFBDIVNUAUO-service-web  ->  service = rails-demo
local RACK_PREFIX = os.getenv("RACK_PREFIX") or ""

local function escape(s)
    return (s:gsub("([%-%.%+%[%]%(%)%$%^%%%?%*])", "%%%1"))
end

function derive_app(tag, ts, record)
    local fam = record["task_definition"]
    if fam == nil then
        return 0, ts, record   -- 0 = leave record unchanged (no metadata to work with)
    end
    local app = fam
    local base = string.match(fam, "^(.-)%-Service")   -- strip "-Service<Kind>-..." suffix
    if base ~= nil then app = base end
    if RACK_PREFIX ~= "" then
        local stripped = string.match(app, "^" .. escape(RACK_PREFIX) .. "%-(.+)$")
        if stripped ~= nil then app = stripped end     -- strip "<rack>-" prefix
    end
    record["service"] = app                            -- Oodle canonical app field
    if record["appname"] ~= nil then
        record["container_id"] = record["appname"]     -- Oodle canonical container id (short)
    end
    return 2, ts, record       -- 2 = record modified
end
convox.yml
environment:
  - OODLE_INSTANCE
  - OODLE_API_KEY
  - RACK_PREFIX
services:
  collector:
    agent:
      enabled: true
      ports:
        - 5140/tcp
    build: .
    scale:
      cpu: 128
      memory: 256
Dockerfile
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable

COPY fluent-bit.conf     /fluent-bit/etc/fluent-bit.conf
COPY parsers-convox.conf /fluent-bit/etc/parsers-convox.conf
COPY derive_app.lua      /fluent-bit/etc/derive_app.lua

Deploy​

1. Protect existing CloudWatch history (do this first)​

Switching an app off Convox's native CloudWatch driver deletes its Convox-managed log group and all history. If you want to keep that history, set DeletionPolicy: Retain on the log group before switching, so the switch orphans it (keeps it in place, same name, all history) instead of deleting it. The log group is managed by the app's CloudFormation stack (Convox names it <rack>-<app>, e.g. gm-test-rails-demo):

# 1. Find the app's CloudFormation stack.
aws cloudformation list-stacks \
  --query "StackSummaries[?contains(StackName,'<app>')].StackName"

# 2. Download its template.
aws cloudformation get-template --stack-name <rack>-<app> \
  --region us-east-1 --query TemplateBody --output json > template.json

# 3. Add "DeletionPolicy": "Retain" to the LogGroup resource in template.json
#    (under Resources.LogGroup), then apply the change, reusing existing params:
aws cloudformation update-stack --stack-name <rack>-<app> --region us-east-1 \
  --template-body file://template.json \
  --parameters ParameterKey=<Key>,UsePreviousValue=true \
  --capabilities CAPABILITY_IAM

After this, the original group is kept in place with all its history, and the agent writes new logs to /convox/<app>.

2. Create the agent app and set credentials​

convox apps create oodle-log-agent -r <rack>

convox env set \
  OODLE_INSTANCE=<instance-id> \
  OODLE_API_KEY=<api-key> \
  RACK_PREFIX=<rack-prefix> \
  -a oodle-log-agent -r <rack>

3. Grant CloudWatch write access (dual-write only)​

convox apps params set IamPolicy=arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
  -a oodle-log-agent -r <rack>

4. Deploy the agent​

Deploy the agent before pointing any app at it. Its syslog driver needs something listening on the host port or its containers fail to start:

convox deploy -a oodle-log-agent -r <rack>

5. Point an app's logs at the agent​

This is per-app and reversible (other apps keep their native CloudWatch driver):

convox apps params set LogDriver=Syslog SyslogDestination=tcp://localhost:5140 SyslogFormat=rfc5424 \
  -a <app> -r <rack>

6. Verify both paths​

Confirm both destinations: the CloudWatch group /convox/<app> is auto-created, and in Oodle the logs are filterable by service=<app>. Allow a few minutes for data to start flowing.

To eyeball enrichment at the record level, uncomment the stdout output in fluent-bit.conf and tail the agent:

convox logs -a oodle-log-agent -r <rack>   # shows enriched records once stdout is enabled

Reverting an app​

To move an app back to Convox's native CloudWatch driver (this restores convox logs -a <app>):

convox apps params set LogDriver=CloudWatch SyslogDestination="" -a <app> -r <rack>

Notes and limitations​

  • EC2 launch type only. The ecs filter's introspection API is not available on Fargate.
  • 172.17.0.1 is the Docker default bridge gateway (stable on the ECS-optimized AMI). If a rack uses a non-default bridge, derive the gateway at start instead of hardcoding.
  • convox scale collector --count 0 does not stop the agent (it's a DaemonSet). Delete the app to actually stop it.
  • Non-rfc5424 lines (occasional Convox framing) fail the parser and are dropped. This is harmless.
  • The debug stdout output is commented out by default; uncomment it while validating to echo enriched records to the agent's own log, and keep it off in production.

Support

If you need assistance or have any questions, please reach out to us through: