Skip to main content

Elastic Container Service (ECS)

Oodle provides a seamless integration with Elastic Container Service and AWS Fargate to ingest logs from your ECS cluster.

Configuration​

To configure Oodle to ingest logs from your ECS cluster, you'll need the following values:

  • OODLE_INSTANCE: Your Oodle instance ID. Go to Settings -> View API Key in your Oodle UI to find out.
  • OODLE_API_KEY: Your Oodle API key for authentication. Go to Settings -> View API Key in your Oodle UI to find out.
  • OODLE_LOGS_HOST: The Oodle Host for log ingestion. Go to Settings -> Logs -> Fluent Bit in your Oodle UI to find out.
  • OODLE_METRICS_HOST: The Oodle Host for metrics ingestion. Go to Settings -> Metrics -> Prometheus in your Oodle UI to find out.

Using AWS FireLens​

If you are using awslogs log driver to send logs to CloudWatch, AWS FireLens is recommended for routing logs to external destinations. Oodle uses Fluent Bit with FireLens for routing logs.

Customer-Hosted Fluent Bit Configs​

It is recommended to host Fluent Bit configs in the customer's S3 buckets as that allows maximum flexibility in terms of being able to version-control the configs, and modifying the config for specific needs. It also simplifies ECS task definitions as config files can be directly pulled from customer's S3 bucket.

  1. Store your fluent bit config in a S3 bucket. Following is an example config which configures dual-write of logs to CloudWatch and Oodle, and also send Fluent Bit internal metrics to Oodle. It also adds following filters:
    • multiline to concatenate partial/split container logs due to container runtime split.
[INPUT]
    name                  fluentbit_metrics
    tag                   internal_metrics
    scrape_interval       2

[FILTER]
    name                  multiline
    match                 *
    multiline.key_content log
    mode                  partial_message

[OUTPUT]
    Name                  cloudwatch_logs
    Match                 *
    region                ${AWS_REGION}
    log_group_name        ${CLOUDWATCH_LOG_GROUP}
    log_stream_name       ${CLOUDWATCH_LOG_STREAM}/${ECS_TASK_ID}
    auto_create_group     true
    Retry_Limit           3

[OUTPUT]
    Name                  http
    Match                 *
    Host                  ${OODLE_LOGS_HOST}
    Port                  443
    URI                   /ingest/v1/logs
    Header                X-OODLE-INSTANCE ${OODLE_INSTANCE}
    Header                X-API-KEY ${OODLE_API_KEY}
    Format                json
    Compress              gzip
    Json_date_key         timestamp
    Json_date_format      iso8601
    Tls                   On
    Retry_Limit           3

[OUTPUT]
    Name                  prometheus_remote_write
    Match                 internal_metrics
    Host                  ${OODLE_METRICS_HOST}
    Port                  443
    Uri                   /v1/prometheus/${OODLE_INSTANCE}/write
    Header                X-API-KEY ${OODLE_API_KEY}
    Log_response_payload  True
    Tls                   On
    # These environment variables are added by aws-for-fluent-bit init container
    add_label             ecs_cluster ${ECS_CLUSTER}
    add_label             ecs_task_arn ${ECS_TASK_ARN}
    add_label             ecs_task_definition ${ECS_TASK_DEFINITION}
    add_label             ecs_task_id ${ECS_TASK_ID}
    Retry_Limit           3
note

Depending on your needs, you can update the fluent bit config, e.g. to only send logs to Oodle, you can remove cloudwatch_logs Output section in the config.

  1. In the ECS task definition, update the value of logDriver from awslogs to awsfirelens for the application container.
{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}
  1. In the ECS task definition, add a Fluent Bit sidecar container to handle log routing. Provide relevant environment variables referenced in the fluent bit config, and provide fluent bit config via aws_fluent_bit_init_s3_1 environment variable.
{
  "name": "fluent-bit",
  "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.32.5.20250305",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_LOGS_HOST",
      "value": "<OODLE_LOGS_HOST>"
    },
    {
      "name": "OODLE_METRICS_HOST",
      "value": "<OODLE_METRICS_HOST>"
    },
    {
      "name": "CLOUDWATCH_LOG_GROUP",
      "value": "<CLOUDWATCH_LOG_GROUP_NAME>"
    },
    {
      "name": "CLOUDWATCH_LOG_STREAM",
      "value": "<CLOUDWATCH_LOG_STREAM_NAME>"
    },
    {
      "name": "aws_fluent_bit_init_s3_1",
      "value": "<arn:aws:s3:::<BUCKET_NAME>/fluent-bit.conf>"
    }
  ],
  "firelensConfiguration": {
    "type": "fluentbit"
  },
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
}
  1. Ensure your ECS task role has permissions to download from S3 bucket.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        }
    ]
}
note

If you want to restrict the permissions to the specific bucket where your Fluent Bit configs are hosted, then you can update Resource to arn:aws:s3:::<BUCKET_NAME>/*

  1. If you are writing logs from Fluent Bit to CloudWatch as well, then ensure your ECS task role has relevant CloudWatch permissions as well.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:<AWS_REGION>:<ACCOUNT_ID>:log-group:<LOG_GROUP_NAME>:*"
        }
    ]
}

Oodle-Hosted Fluent Bit Configs​

If you prefer to use Oodle-Hosted Fluent Bit configs, then you can follow steps in this section.

Send to CloudWatch and Oodle​

In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.

  1. In the application container definition, update the value of logDriver from awslogs to awsfirelens.
{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}
  1. Add an ephemeral volume in the ECS task definition. This volume will be used to store Fluent Bit configuration file as configured in Step #3.
"volumes": [
  {
    "name": "config",
    "host": {}
  }
]
  1. Add two containers to the ECS task definition:
    • An init container to download Fluent Bit configuration file
    • A Fluent Bit sidecar container to handle log routing to both CloudWatch and Oodle.
{
  "name": "config-init",
  "image": "public.ecr.aws/docker/library/alpine:3.21.3",
  "essential": false,
  "memoryReservation": 64,
  "command": [
    "sh",
    "-c",
    "set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-cloudwatch-v3.conf || exit 1"
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": false
    }
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
},
{
  "name": "fluent-bit",
  "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.32.5.20250305",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_LOGS_HOST",
      "value": "<OODLE_LOGS_HOST>"
    },
    {
      "name": "OODLE_METRICS_HOST",
      "value": "<OODLE_METRICS_HOST>"
    },
    {
      "name": "CLOUDWATCH_LOG_GROUP",
      "value": "<CLOUDWATCH_LOG_GROUP_NAME>"
    },
    {
      "name": "CLOUDWATCH_LOG_STREAM",
      "value": "<CLOUDWATCH_LOG_STREAM_NAME>"
    }
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": true
    }
  ],
  "dependsOn": [
    {
      "containerName": "config-init",
      "condition": "COMPLETE"
    }
  ],
  "firelensConfiguration": {
    "type": "fluentbit",
    "options": {
      "config-file-type": "file",
      "config-file-value": "/oodle/fluent-bit.conf"
    }
  },
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
}
  1. Ensure your ECS task role) has relevant CloudWatch permissions.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:<AWS_REGION>:<ACCOUNT_ID>:log-group:<LOG_GROUP_NAME>:*"
        }
    ]
}
Send to Oodle Only​

In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.

  1. Configure the application container and volume (Steps 1-2 above).
  2. Update the config-init container definition.
{
  "name": "config-init",
  "image": "public.ecr.aws/docker/library/alpine:3.21.3",
  "essential": false,
  "memoryReservation": 64,
  "command": [
    "sh",
    "-c",
    "set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-v3.conf || exit 1"
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": false
    }
  ],
    "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
}

The config-init container downloads a simplified configuration file from fluent-bit-v3.conf that routes logs exclusively to Oodle.

In addition, environment variables CLOUDWATCH_LOG_GROUP and CLOUDWATCH_LOG_STREAM can be removed from the fluent-bit container.

Optional Configurations​

This section documents configurations which can be used if you have the specified use-cases.

JSON Logs​

If your applications emit logs in json format, then you can add a fluent parser to parse the json logs before routing them to destinations. It makes use of multi-config-support and parse-json capability in AWS for Fluent Bit image.

To add parse-json parser in your fluent-bit configuration, you can add following environment variable to fluent-bit container:

    {
      "name": "aws_fluent_bit_init_file_1",
      "value": "/fluent-bit/configs/parse-json.conf"
    }

The default parser used in AWS for Fluent Bit image uses Time_Format %d/%b/%Y:%H:%M:%S %z to parse the timestamp from the logs. If your application logs have a different timestamp format, you can override the json parser. As an example, if your timestamp format is ISO8601 (%Y-%m-%dT%H:%M:%S.%LZ), then you can follow below steps:

  1. Update config-init container command to download custom json parser related configuration files, add following wget commands in the command used in config-init container:
wget -O /oodle/parser.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-parser-v1.conf && wget -O /oodle/parse-json.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-filter-parse-json-iso8601-v1.conf
  1. Add following environment variables in fluent-bit container:
{
    "name": "aws_fluent_bit_init_file_1",
    "value": "/oodle/parser.conf"
},
{
    "name": "aws_fluent_bit_init_file_2",
    "value": "/oodle/parse-json.conf"
}
note

Parser definitions needs to be provided as separate file and cannot be merged with Filter definitions in a single file.

You can choose to store these config files in your S3 bucket as well, and use aws_fluent_bit_init_s3_<n> environment variables to refer to them.

Concatenate Partial/Split Container Logs​

Container runtime splits log lines larger than 16KB into multiple log lines. Fluent Bit can concatenate these split log lines using multiline filter. To use it, add following filter in your fluent bit configuration:

[FILTER]
    name                  multiline
    match                 *
    multiline.key_content log
    mode                  partial_message
note

This needs to be first filter in the pipeline so that all remaining filters operate on concatenated log lines.

Enrich log lines with additional key-value pair​

Say, you want to ingest your dev and prod cluster logs to Oodle. You can enrich each log line with a environment: dev or environment: prod field by making following changes:

  1. Update config-init container command to download custom modify filter configuration file, add following wget command in the command used in config-init container:
wget -O /oodle/modify-add-field.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-modify-add-field-v1.conf
  1. Add following environment variables in fluent-bit container:
{
    "name": "FIELD_NAME",
    "value": "environment"
},
{
    "name": "FIELD_VALUE",
    "value": "dev"
},
{
    "name": "aws_fluent_bit_init_file_1",
    "value": "/oodle/modify-add-field.conf"
},
note

The numbering in aws_fluent_bit_init_file_* can be used to add multiple init files, e.g. if you need both json parsing and enrich field, then you can add all files needed with increasing counter.

Using OpenTelemetry Collector​

Users can run OpenTelemetry (OTel) Collector contrib image to send ECS logs to Oodle.

note
  1. AWS Distro for OpenTelemetry image is not supported as it doesn't contain the Fluent Forward Receiver required by this configuration.
  2. Fluent Forward Receiver is currenlty in beta status.

Send to CloudWatch and Oodle​

In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.

  1. In the application container definition, update the value of logDriver from awslogs to awsfirelens.
{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}

Fluent Forward Receiver in OTel collector is used in this configuration. Therefore, the value of logDriver is set to awsfirelens and the firelensConfiguration type in next step is set to fluentbit.

  1. Add an OTel collector sidecar container in the ECS task definiton.
{
  "name": "otel-collector",
  "image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_LOGS_HOST",
      "value": "<OODLE_LOGS_HOST>"
    },
    {
      "name": "CLOUDWATCH_LOG_GROUP",
      "value": "<CLOUDWATCH_LOG_GROUP_NAME>"
    },
    {
      "name": "CLOUDWATCH_LOG_STREAM",
      "value": "<CLOUDWATCH_LOG_STREAM_NAME>"
    }
  ],
  "command": [
    "--config",
    "https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-cloudwatch-v2.yaml"
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  },
  "firelensConfiguration": {
    "type": "fluentbit"
  }
}
  1. Ensure your ECS task role) has relevant CloudWatch permissions.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:<AWS_REGION>:<ACCOUNT_ID>:log-group:<LOG_GROUP_NAME>:*"
        }
    ]
}

OTeL collector downloads an otel-config-cloudwatch-v2.yaml from a public S3 repository owned by Oodle. The config files in this repository are versioned and immutable.

Send to Oodle Only​

In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.

  1. Configure the application container (Step 1 above).
  2. Update the otel-collector container definition.
{
  "name": "otel-collector",
  "image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_LOGS_HOST",
      "value": "<OODLE_LOGS_HOST>"
    }
  ],
  "command": [
    "--config",
    "https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-v2.yaml"
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  },
  "firelensConfiguration": {
    "type": "fluentbit"
  }
}

The otel-collector container downloads a simplified configuration file from otel-config-v2.yaml that routes logs exclusively to Oodle.

In addition, environment variables CLOUDWATCH_LOG_GROUP and CLOUDWATCH_LOG_STREAM are also removed from the otel-collector container.

CloudWatch Logs Integration​

If you are already sending your ECS logs to CloudWatch and prefer to push logs from CloudWatch to Oodle instead of modifying your ECS task definitions, you can use our CloudWatch integration. This approach allows you to keep your existing logging setup while still getting your logs into Oodle.

For detailed instructions on how to set up the CloudWatch to Oodle integration, please refer to our CloudWatch integration guide.

Support​

If you need assistance or have any questions, please reach out to us through:

  • The help chat widget in the bottom-right corner of this page
  • Email at support@oodle.ai