Skip to main content

Amazon S3

Oodle provides seamless integration with Amazon S3 for ingesting logs stored in S3 buckets. This is useful for logs that are written directly to S3, such as ALB access logs, CloudFront logs, or any application logs stored in S3.

The integration uses an SQS queue to receive S3 event notifications when new objects are created. Oodle then reads the log files from S3 and ingests them.

Configuration​

Prerequisites​

Before setting up the integration, you'll need the following parameters from Oodle:

  • OodleAWSAccountId: Oodle's AWS account ID
  • ExternalId: A unique external ID provided by Oodle for secure cross-account access

Contact the Oodle team to obtain these values.

Step 1: Deploy CloudFormation Stack​

Deploy the CloudFormation stack that creates the necessary IAM role and SQS queue for S3 log ingestion.

Use the Quick Create link to create the CloudFormation stack in your AWS account. Be sure to choose the appropriate AWS region where your S3 buckets are located.

You will be asked to provide the following parameters:

ParameterDescription
OodleAWSAccountIdOodle's AWS account ID (provided by Oodle)
ExternalIdExternal ID for secure role assumption (provided by Oodle)
S3BucketPattern1First S3 bucket name with optional prefix (required). Example: my-logs-bucket or my-logs-bucket/alb-logs/
S3BucketPattern2-5Additional S3 bucket names with optional prefixes (optional)

The CloudFormation stack creates the following resources:

  1. SQS Queue (OodleS3NotificationQueue-<stack-name>): Receives S3 event notifications when new log files are created
  2. IAM Role (OodleIntegrationS3LogIngestion-<stack-name>): Allows Oodle to assume this role to read logs from S3 and receive messages from the SQS queue

Step 2: Share Stack Outputs with Oodle​

After the CloudFormation stack is created:

  1. Navigate to the CloudFormation console
  2. Select the stack you created (e.g., "OodleS3Logs")
  3. Click on the Outputs tab
  4. Share the following values with the Oodle team:
    • RoleARN: ARN of the IAM role
    • QueueARN: ARN of the SQS queue

Step 3: Configure S3 Event Notifications​

For each S3 bucket containing logs, you need to create an event notification to send messages to the SQS queue when new objects are created.

  1. Open the S3 console
  2. Select the bucket containing your logs
  3. Navigate to the Properties tab
S3 bucket properties
  1. Scroll to the Event notifications section and click Create event notification
Create event notification
  1. Configure the notification:
    • Event name: Enter a descriptive name (e.g., "Oodle Logs")
    • Prefix (optional): Add a path prefix if logs are in a specific directory
    • Event types: Select All object create events
    Configure notifications
    • Destination: Choose SQS queue
    • SQS queue: Select Choose from your SQS queues
    • Queue: Select OodleS3NotificationQueue-<stack-name>
  2. Click Save changes
Configure SQS notifications

Repeat these steps for all buckets containing logs you want to ingest.

Step 4: Verify the Integration​

To verify that notifications are being sent to the SQS queue:

  1. Open the SQS console
  2. Search for OodleS3NotificationQueue-<stack-name>
  3. Check that Messages available shows a value greater than 0
SQS Verification

This confirms that S3 is successfully sending notifications when new log files are created. Once Oodle team configures ingestion pipeline, logs would be visible in Oodle UI.

Support​

If you need assistance or have any questions, please reach out to us through:

  • The help chat widget in the bottom-right corner of this page
  • Email at support@oodle.ai