Skip to main content

Elastic Container Service (ECS)

Oodle provides a seamless integration with Elastic Container Service and AWS Fargate to ingest logs from your ECS cluster.

Configuration​

To configure Oodle to ingest logs from your ECS cluster, you'll need the following values:

  • OODLE_INSTANCE: Your Oodle instance ID
  • OODLE_API_KEY: Your Oodle API key for authentication
  • OODLE_ENDPOINT: The Oodle endpoint URL for log ingestion

Using AWS FireLens​

If you are using awslogs log driver to send logs to CloudWatch, AWS FireLens is recommended for routing logs to external destinations. Oodle uses Fluent Bit with FireLens for routing logs.

Send to CloudWatch and Oodle​

In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.

  1. In the application container definition, update the value of logDriver from awslogs to awsfirelens.
note

Ensure the ECS task role has the sufficient permissions to send logs to CloudWatch. See IAM Permissions.

{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}
  1. Add an ephemeral volume in the ECS task definition. This volume will be used to store Fluent Bit configuration file as configured in Step #3.
"volumes": [
  {
    "name": "config",
    "host": {}
  }
]
  1. Add two containers to the ECS task definition:
    • An init container to download Fluent Bit configuration file
    • A Fluent Bit sidecar container to handle log routing to both CloudWatch and Oodle.
{
  "name": "config-init",
  "image": "public.ecr.aws/docker/library/alpine:3.21.3",
  "essential": false,
  "memoryReservation": 64,
  "command": [
    "sh",
    "-c",
    "set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-cloudwatch-v1.conf || exit 1"
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": false
    }
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
},
{
  "name": "fluent-bit",
  "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.32.5.20250305",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_ENDPOINT",
      "value": "<OODLE_ENDPOINT>"
    },
    {
      "name": "CLOUDWATCH_LOG_GROUP",
      "value": "<CLOUDWATCH_LOG_GROUP_NAME>"
    },
    {
      "name": "CLOUDWATCH_LOG_STREAM",
      "value": "<CLOUDWATCH_LOG_STREAM_NAME>"
    },
    {
      "name": "aws_fluent_bit_init_file_1",
      "value": "/fluent-bit/configs/parse-json.conf"
    }
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": true
    }
  ],
  "dependsOn": [
    {
      "containerName": "config-init",
      "condition": "COMPLETE"
    }
  ],
  "firelensConfiguration": {
    "type": "fluentbit",
    "options": {
      "config-file-type": "file",
      "config-file-value": "/oodle/fluent-bit.conf"
    }
  },
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
}
note

We add a fluent parser to parse the json logs before routing them to destinations. It makes use of multi-config-support and parse-json capability in AWS for Fluent Bit image.

Send to Oodle Only​

In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.

  1. Configure the application container and volume (Steps 1-2 above).
  2. Update the config-init container definition.
{
  "name": "config-init",
  "image": "public.ecr.aws/docker/library/alpine:3.21.3",
  "essential": false,
  "memoryReservation": 64,
  "command": [
    "sh",
    "-c",
    "set -e && apk add --no-cache ca-certificates wget && wget -O /oodle/fluent-bit.conf https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/fluent-bit/fluent-bit-v1.conf || exit 1"
  ],
  "mountPoints": [
    {
      "sourceVolume": "config",
      "containerPath": "/oodle",
      "readOnly": false
    }
  ],
    "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  }
}

The config-init container downloads a simplified configuration file from fluent-bit-v1.conf that routes logs exclusively to Oodle.

In addition, environment variables CLOUDWATCH_LOG_GROUP and CLOUDWATCH_LOG_STREAM can be removed from the fluent-bit container.

Using OpenTelemetry Collector​

Users can run OpenTelemetry (OTel) Collector contrib image to send ECS logs to Oodle.

note
  1. AWS Distro for OpenTelemetry image is not supported as it doesn't contain the Fluent Forward Receiver required by this configuration.
  2. Fluent Forward Receiver is currenlty in beta status.

Send to CloudWatch and Oodle​

In this configuration, ECS application container logs are sent to both CloudWatch and Oodle. Sidecar containers logs are sent to CloudWatch.

  1. In the application container definition, update the value of logDriver from awslogs to awsfirelens.
note

Ensure the ECS task role has the sufficient permissions to send logs to CloudWatch. See IAM Permissions.

{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}

Fluent Forward Receiver in OTel collector is used in this configuration. Therefore, the value of logDriver is set to awsfirelens and the firelensConfiguration type in next step is set to fluentbit.

  1. Add an OTel collector sidecar container in the ECS task definiton.
{
  "name": "otel-collector",
  "image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_ENDPOINT",
      "value": "<OODLE_ENDPOINT>"
    },
    {
      "name": "CLOUDWATCH_LOG_GROUP",
      "value": "<CLOUDWATCH_LOG_GROUP_NAME>"
    },
    {
      "name": "CLOUDWATCH_LOG_STREAM",
      "value": "<CLOUDWATCH_LOG_STREAM_NAME>"
    }
  ],
  "command": [
    "--config",
    "https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-cloudwatch-v1.yaml"
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  },
  "firelensConfiguration": {
    "type": "fluentbit"
  }
}

OTeL collector downloads an otel-config-cloudwatch-v1.yaml from a public S3 repository owned by Oodle. The config files in this repository are versioned and immutable.

Send to Oodle Only​

In this configuration, ECS application container logs are sent to Oodle only. Sidecar containers logs are sent to CloudWatch.

  1. Configure the application container (Step 1 above).
  2. Update the otel-collector container definition.
{
  "name": "otel-collector",
  "image": "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.121.0",
  "essential": false,
  "memory": 200,
  "environment": [
    {
      "name": "OODLE_INSTANCE",
      "value": "<OODLE_INSTANCE>"
    },
    {
      "name": "OODLE_API_KEY",
      "value": "<OODLE_API_KEY>"
    },
    {
      "name": "OODLE_ENDPOINT",
      "value": "<OODLE_ENDPOINT>"
    }
  ],
  "command": [
    "--config",
    "https://oodle-configs.s3.us-west-2.amazonaws.com/logs/ecs/otel/otel-config-v1.yaml"
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-region": "<AWS_REGION>",
      "awslogs-group": "<CLOUDWATCH_LOG_GROUP_NAME>",
      "awslogs-stream-prefix": "<CLOUDWATCH_LOG_STREAM_PREFIX>"
    }
  },
  "firelensConfiguration": {
    "type": "fluentbit"
  }
}

The otel-collector container downloads a simplified configuration file from otel-config-v1.yaml that routes logs exclusively to Oodle.

In addition, environment variables CLOUDWATCH_LOG_GROUP and CLOUDWATCH_LOG_STREAM are also removed from the otel-collector container.

Common Issues​

IAM Permissions​

If you encounter issues with CloudWatch log delivery, it might be due to insufficient IAM permissions. Ensure your ECS task role has the following minimum permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:<AWS_REGION>:<ACCOUNT_ID>:log-group:<LOG_GROUP_NAME>:*"
        }
    ]
}

Support​

If you need assistance or have any questions, please reach out to us through:

  • The help chat widget in the bottom-right corner of this page
  • Email at support@oodle.ai