Elastic Container Service (ECS)

Oodle provides a seamless integration with Elastic Container Service and AWS Fargate to ingest logs from your ECS cluster.

Configuration

To configure Oodle to ingest logs from your ECS cluster, you'll need the following values:

OODLE_INSTANCE: Your Oodle instance ID
OODLE_API_KEY: Your Oodle API key for authentication
OODLE_ENDPOINT: The Oodle endpoint URL for log ingestion

Using AWS FireLens

If you are using awslogs log driver to send logs to CloudWatch, AWS FireLens is recommended for routing logs to external destinations. Oodle uses Fluent Bit with FireLens for routing logs.

Send to CloudWatch and Oodle

In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.

In the application container definition, update the value of logDriver from awslogs to awsfirelens.

note

Ensure the ECS task role has the sufficient permissions to send logs to CloudWatch. See IAM Permissions.

{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}

Add an ephemeral volume in the ECS task definition. This volume will be used to store Fluent Bit configuration file as configured in Step #3.

"volumes": [
  {
    "name": "config",
    "host": {}
  }
]

Add two containers to the ECS task definition:
- An init container to download Fluent Bit configuration file
- A Fluent Bit sidecar container to handle log routing to both CloudWatch and Oodle.

{
  "name": "config-init",
  "image": "public.ecr.aws/docker/library/alpine:3.21.3",
  "essential": false,
  "memoryReservation": 64,
  "command": [
    "sh",
    "-c",
    "set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-cloudwatch-v1.conf || exit 1"
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": false
    }
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
},
{
  "name": "fluent-bit",
  "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.32.5.20250305",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_ENDPOINT",
      "value": "<OODLE_ENDPOINT>"
    },
    {
      "name": "CLOUDWATCH_LOG_GROUP",
      "value": "<CLOUDWATCH_LOG_GROUP_NAME>"
    },
    {
      "name": "CLOUDWATCH_LOG_STREAM",
      "value": "<CLOUDWATCH_LOG_STREAM_NAME>"
    }
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": true
    }
  ],
  "dependsOn": [
    {
      "containerName": "config-init",
      "condition": "COMPLETE"
    }
  ],
  "firelensConfiguration": {
    "type": "fluentbit",
    "options": {
      "config-file-type": "file",
      "config-file-value": "/oodle/fluent-bit.conf"
    }
  },
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
}

Send to Oodle Only

In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.

Configure the application container and volume (Steps 1-2 above).
Update the config-init container definition.

{
  "name": "config-init",
  "image": "public.ecr.aws/docker/library/alpine:3.21.3",
  "essential": false,
  "memoryReservation": 64,
  "command": [
    "sh",
    "-c",
    "set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-v1.conf || exit 1"
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": false
    }
  ],
    "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
}

The config-init container downloads a simplified configuration file from fluent-bit-v1.conf that routes logs exclusively to Oodle.

In addition, environment variables CLOUDWATCH_LOG_GROUP and CLOUDWATCH_LOG_STREAM can be removed from the fluent-bit container.

Optional Configurations

This section documents configurations which can be used if you have the specified use-cases.

JSON Logs

If your applications emit logs in json format, then you can add a fluent parser to parse the json logs before routing them to destinations. It makes use of multi-config-support and parse-json capability in AWS for Fluent Bit image.

To add parse-json parser in your fluent-bit configuration, you can add following environment variable to fluent-bit container:

    {
      "name": "aws_fluent_bit_init_file_1",
      "value": "/fluent-bit/configs/parse-json.conf"
    }

The default parser used in AWS for Fluent Bit image uses Time_Format %d/%b/%Y:%H:%M:%S %z to parse the timestamp from the logs. If your application logs have a different timestamp format, you can override the json parser. As an example, if your timestamp format is ISO8601 (%Y-%m-%dT%H:%M:%S.%LZ), then you can follow below steps:

Update config-init container command to download custom json parser related configuration files, add following wget commands in the command used in config-init container:

wget -O /oodle/parser.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-parser-v1.conf && wget -O /oodle/parse-json.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-filter-parse-json-iso8601-v1.conf

Add following environment variables in fluent-bit container:

{
    "name": "aws_fluent_bit_init_file_1",
    "value": "/oodle/parser.conf"
},
{
    "name": "aws_fluent_bit_init_file_2",
    "value": "/oodle/parse-json.conf"
}

Enrich log lines with additional key-value pair

Say, you want to ingest your dev and prod cluster logs to Oodle. You can enrich each log line with a environment: dev or environment: prod field by making following changes:

Update config-init container command to download custom modify filter configuration file, add following wget command in the command used in config-init container:

wget -O /oodle/modify-add-field.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-modify-add-field-v1.conf

Add following environment variables in fluent-bit container:

{
    "name": "FIELD_NAME",
    "value": "environment"
},
{
    "name": "FIELD_VALUE",
    "value": "dev"
},
{
    "name": "aws_fluent_bit_init_file_1",
    "value": "/oodle/modify-add-field.conf"
},

note

The numbering in aws_fluent_bit_init_file_* can be used to add multiple init files, e.g. if you need both json parsing and enrich field, then you can add all files needed with increasing counter.

Using OpenTelemetry Collector

Users can run OpenTelemetry (OTel) Collector contrib image to send ECS logs to Oodle.

note

AWS Distro for OpenTelemetry image is not supported as it doesn't contain the Fluent Forward Receiver required by this configuration.
Fluent Forward Receiver is currenlty in beta status.

Send to CloudWatch and Oodle

In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.

In the application container definition, update the value of logDriver from awslogs to awsfirelens.

note

Ensure the ECS task role has the sufficient permissions to send logs to CloudWatch. See IAM Permissions.

{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}

Fluent Forward Receiver in OTel collector is used in this configuration. Therefore, the value of logDriver is set to awsfirelens and the firelensConfiguration type in next step is set to fluentbit.

Add an OTel collector sidecar container in the ECS task definiton.

{
  "name": "otel-collector",
  "image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_ENDPOINT",
      "value": "<OODLE_ENDPOINT>"
    },
    {
      "name": "CLOUDWATCH_LOG_GROUP",
      "value": "<CLOUDWATCH_LOG_GROUP_NAME>"
    },
    {
      "name": "CLOUDWATCH_LOG_STREAM",
      "value": "<CLOUDWATCH_LOG_STREAM_NAME>"
    }
  ],
  "command": [
    "--config",
    "https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-cloudwatch-v1.yaml"
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  },
  "firelensConfiguration": {
    "type": "fluentbit"
  }
}

OTeL collector downloads an otel-config-cloudwatch-v1.yaml from a public S3 repository owned by Oodle. The config files in this repository are versioned and immutable.

Send to Oodle Only

In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.

Configure the application container (Step 1 above).
Update the otel-collector container definition.

{
  "name": "otel-collector",
  "image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_ENDPOINT",
      "value": "<OODLE_ENDPOINT>"
    }
  ],
  "command": [
    "--config",
    "https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-v1.yaml"
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  },
  "firelensConfiguration": {
    "type": "fluentbit"
  }
}

The otel-collector container downloads a simplified configuration file from otel-config-v1.yaml that routes logs exclusively to Oodle.

In addition, environment variables CLOUDWATCH_LOG_GROUP and CLOUDWATCH_LOG_STREAM are also removed from the otel-collector container.

Common Issues

IAM Permissions

If you encounter issues with CloudWatch log delivery, it might be due to insufficient IAM permissions. Ensure your ECS task role has the following minimum permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:<AWS_REGION>:<ACCOUNT_ID>:log-group:<LOG_GROUP_NAME>:*"
        }
    ]
}

CloudWatch Logs Integration

If you are already sending your ECS logs to CloudWatch and prefer to push logs from CloudWatch to Oodle instead of modifying your ECS task definitions, you can use our CloudWatch integration. This approach allows you to keep your existing logging setup while still getting your logs into Oodle.

For detailed instructions on how to set up the CloudWatch to Oodle integration, please refer to our CloudWatch integration guide.

Support

If you need assistance or have any questions, please reach out to us through:

The help chat widget in the bottom-right corner of this page
Email at support@oodle.ai

Configuration​

Using AWS FireLens​

Send to CloudWatch and Oodle​

Send to Oodle Only​

Optional Configurations​

JSON Logs​

Enrich log lines with additional key-value pair​

Using OpenTelemetry Collector​

Send to CloudWatch and Oodle​

Send to Oodle Only​

Common Issues​

IAM Permissions​

CloudWatch Logs Integration​

Support​

Configuration

Using AWS FireLens

Send to CloudWatch and Oodle

Send to Oodle Only

Optional Configurations

JSON Logs

Enrich log lines with additional key-value pair

Using OpenTelemetry Collector

Send to CloudWatch and Oodle

Send to Oodle Only

Common Issues

IAM Permissions

CloudWatch Logs Integration

Support