Elastic Container Service (ECS)
Oodle provides a seamless integration with Elastic Container Service and AWS Fargate to ingest logs from your ECS cluster.
Configuration
To configure Oodle to ingest logs from your ECS cluster, you'll need the following values:
OODLE_INSTANCE
: Your Oodle instance IDOODLE_API_KEY
: Your Oodle API key for authenticationOODLE_ENDPOINT
: The Oodle endpoint URL for log ingestion
Using AWS FireLens
If you are using awslogs log driver to send logs to CloudWatch, AWS FireLens is recommended for routing logs to external destinations. Oodle uses Fluent Bit with FireLens for routing logs.
Send to CloudWatch and Oodle
In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.
- In the application container definition, update the value of
logDriver
fromawslogs
toawsfirelens
.
Ensure the ECS task role has the sufficient permissions to send logs to CloudWatch. See IAM Permissions.
{
"logConfiguration": {
"logDriver": "awsfirelens"
}
}
- Add an ephemeral volume in the ECS task definition. This volume will be used to store Fluent Bit configuration file as configured in Step #3.
"volumes": [
{
"name": "config",
"host": {}
}
]
- Add two containers to the ECS task definition:
- An init container to download Fluent Bit configuration file
- A Fluent Bit sidecar container to handle log routing to both CloudWatch and Oodle.
{
"name": "config-init",
"image": "public.ecr.aws/docker/library/alpine:3.21.3",
"essential": false,
"memoryReservation": 64,
"command": [
"sh",
"-c",
"set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-cloudwatch-v1.conf || exit 1"
],
"mountPoints": [
{
"sourceVolume": "config",
"containerPath": "/oodle",
"readOnly": false
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "<AWS_REGION>",
"awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
"awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
}
}
},
{
"name": "fluent-bit",
"image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.32.5.20250305",
"essential": false,
"memory": 200,
"environment": [
{
"name": "OODLE_INSTANCE",
"value": "<OODLE_INSTANCE>"
},
{
"name": "OODLE_API_KEY",
"value": "<OODLE_API_KEY>"
},
{
"name": "OODLE_ENDPOINT",
"value": "<OODLE_ENDPOINT>"
},
{
"name": "CLOUDWATCH_LOG_GROUP",
"value": "<CLOUDWATCH_LOG_GROUP_NAME>"
},
{
"name": "CLOUDWATCH_LOG_STREAM",
"value": "<CLOUDWATCH_LOG_STREAM_NAME>"
}
],
"mountPoints": [
{
"sourceVolume": "config",
"containerPath": "/oodle",
"readOnly": true
}
],
"dependsOn": [
{
"containerName": "config-init",
"condition": "COMPLETE"
}
],
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/oodle/fluent-bit.conf"
}
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "<AWS_REGION>",
"awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
"awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
}
}
}
Send to Oodle Only
In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.
- Configure the application container and volume (Steps 1-2 above).
- Update the
config-init
container definition.
{
"name": "config-init",
"image": "public.ecr.aws/docker/library/alpine:3.21.3",
"essential": false,
"memoryReservation": 64,
"command": [
"sh",
"-c",
"set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-v1.conf || exit 1"
],
"mountPoints": [
{
"sourceVolume": "config",
"containerPath": "/oodle",
"readOnly": false
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "<AWS_REGION>",
"awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
"awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
}
}
}
The config-init
container downloads a simplified configuration file from fluent-bit-v1.conf
that routes logs exclusively to Oodle.
In addition, environment variables CLOUDWATCH_LOG_GROUP
and CLOUDWATCH_LOG_STREAM
can be removed from
the fluent-bit
container.
Optional Configurations
This section documents configurations which can be used if you have the specified use-cases.
JSON Logs
If your applications emit logs in json format, then you can add a fluent parser to parse the json logs before routing them to destinations. It makes use of multi-config-support and parse-json capability in AWS for Fluent Bit image.
To add parse-json
parser in your fluent-bit
configuration, you can add following environment variable to fluent-bit
container:
{
"name": "aws_fluent_bit_init_file_1",
"value": "/fluent-bit/configs/parse-json.conf"
}
The default parser used in AWS for Fluent Bit image uses Time_Format %d/%b/%Y:%H:%M:%S %z
to parse the timestamp from
the logs. If your application logs have a different timestamp format, you can override the json parser. As an example,
if your timestamp format is ISO8601 (%Y-%m-%dT%H:%M:%S.%LZ
), then you can follow below steps:
- Update
config-init
container command to download custom json parser related configuration files, add followingwget
commands in the command used inconfig-init
container:
wget -O /oodle/parser.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-parser-v1.conf && wget -O /oodle/parse-json.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-filter-parse-json-iso8601-v1.conf
- Add following environment variables in
fluent-bit
container:
{
"name": "aws_fluent_bit_init_file_1",
"value": "/oodle/parser.conf"
},
{
"name": "aws_fluent_bit_init_file_2",
"value": "/oodle/parse-json.conf"
}
Enrich log lines with additional key-value pair
Say, you want to ingest your dev and prod cluster logs to Oodle. You can enrich each log line with a environment: dev
or environment: prod
field by making following changes:
- Update
config-init
container command to download custom modify filter configuration file, add followingwget
command in the command used inconfig-init
container:
wget -O /oodle/modify-add-field.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-modify-add-field-v1.conf
- Add following environment variables in
fluent-bit
container:
{
"name": "FIELD_NAME",
"value": "environment"
},
{
"name": "FIELD_VALUE",
"value": "dev"
},
{
"name": "aws_fluent_bit_init_file_1",
"value": "/oodle/modify-add-field.conf"
},
The numbering in aws_fluent_bit_init_file_*
can be used to add multiple init files, e.g. if you need both
json parsing and enrich field, then you can add all files needed with increasing counter.
Using OpenTelemetry Collector
Users can run OpenTelemetry (OTel) Collector contrib image to send ECS logs to Oodle.
- AWS Distro for OpenTelemetry image is not supported as it doesn't contain the Fluent Forward Receiver required by this configuration.
- Fluent Forward Receiver is currenlty in beta status.
Send to CloudWatch and Oodle
In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.
- In the application container definition, update the value of
logDriver
fromawslogs
toawsfirelens
.
Ensure the ECS task role has the sufficient permissions to send logs to CloudWatch. See IAM Permissions.
{
"logConfiguration": {
"logDriver": "awsfirelens"
}
}
Fluent Forward Receiver in OTel collector is used in this configuration. Therefore, the value of logDriver
is set to awsfirelens
and the firelensConfiguration
type in next step is set to fluentbit
.
- Add an OTel collector sidecar container in the ECS task definiton.
{
"name": "otel-collector",
"image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
"essential": false,
"memory": 200,
"environment": [
{
"name": "OODLE_INSTANCE",
"value": "<OODLE_INSTANCE>"
},
{
"name": "OODLE_API_KEY",
"value": "<OODLE_API_KEY>"
},
{
"name": "OODLE_ENDPOINT",
"value": "<OODLE_ENDPOINT>"
},
{
"name": "CLOUDWATCH_LOG_GROUP",
"value": "<CLOUDWATCH_LOG_GROUP_NAME>"
},
{
"name": "CLOUDWATCH_LOG_STREAM",
"value": "<CLOUDWATCH_LOG_STREAM_NAME>"
}
],
"command": [
"--config",
"https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-cloudwatch-v1.yaml"
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "<AWS_REGION>",
"awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
"awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
}
},
"firelensConfiguration": {
"type": "fluentbit"
}
}
OTeL collector downloads an otel-config-cloudwatch-v1.yaml from a public S3 repository owned by Oodle. The config files in this repository are versioned and immutable.
Send to Oodle Only
In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.
- Configure the application container (Step 1 above).
- Update the
otel-collector
container definition.
{
"name": "otel-collector",
"image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
"essential": false,
"memory": 200,
"environment": [
{
"name": "OODLE_INSTANCE",
"value": "<OODLE_INSTANCE>"
},
{
"name": "OODLE_API_KEY",
"value": "<OODLE_API_KEY>"
},
{
"name": "OODLE_ENDPOINT",
"value": "<OODLE_ENDPOINT>"
}
],
"command": [
"--config",
"https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-v1.yaml"
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "<AWS_REGION>",
"awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
"awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
}
},
"firelensConfiguration": {
"type": "fluentbit"
}
}
The otel-collector
container downloads a simplified configuration file from otel-config-v1.yaml that routes logs exclusively to Oodle.
In addition, environment variables CLOUDWATCH_LOG_GROUP
and CLOUDWATCH_LOG_STREAM
are also removed from the otel-collector
container.
Common Issues
IAM Permissions
If you encounter issues with CloudWatch log delivery, it might be due to insufficient IAM permissions. Ensure your ECS task role has the following minimum permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:<AWS_REGION>:<ACCOUNT_ID>:log-group:<LOG_GROUP_NAME>:*"
}
]
}
CloudWatch Logs Integration
If you are already sending your ECS logs to CloudWatch and prefer to push logs from CloudWatch to Oodle instead of modifying your ECS task definitions, you can use our CloudWatch integration. This approach allows you to keep your existing logging setup while still getting your logs into Oodle.
For detailed instructions on how to set up the CloudWatch to Oodle integration, please refer to our CloudWatch integration guide.
Support
If you need assistance or have any questions, please reach out to us through:
- The help chat widget in the bottom-right corner of this page
- Email at support@oodle.ai