Bring Your Own Bucket

With Bring Your Own Bucket, Oodle keeps the metrics and traces of your Oodle instance in an Amazon S3 bucket in your own AWS account. You own the bucket, and you control access to it through an IAM role that Oodle assumes.

Which data goes to your bucket

After the setup, Oodle writes this data to your bucket:

• Metrics
• Traces

Oodle writes only new data to your bucket. Data that Oodle wrote before the setup stays in the Oodle bucket, and you can continue to query it. Queries read from the two buckets and show one result.

Requirements

• Bring Your Own Bucket enabled for your instance. To enable it, contact Oodle support.
• An Oodle Admin role. Only an admin can see and change the bucket settings.
• An AWS account with permissions to create CloudFormation stacks, S3 buckets, and IAM roles.

Setup

1. In Oodle, go to Settings and open Bring Your Own Bucket in the Admin section.
2. Click Launch CloudFormation stack. The AWS console opens the stack in the region of your Oodle deployment, with all the required parameters already set. The stack fails in other regions, because Oodle cannot use a bucket in another region.
3. Optional: choose the Encryption before you launch the stack. The default, S3 managed keys (SSE-S3), needs no change. To use a KMS key, see Encrypt with a KMS key.
4. Create the stack. The stack creates:
   • An S3 bucket with public access blocked, default encryption (SSE-S3, or SSE-KMS with the S3 Bucket Key), and the Oodle lifecycle rules.
   • An IAM role named OodleS3Integration-BYOB-<suffix> that Oodle can assume with the external ID.
   • For SSE-KMS with no key ARN, a KMS key.
5. When the stack status is CREATE_COMPLETE, open its Outputs tab. Copy the BucketName and RoleARN values into the settings page. For SSE-KMS, also copy the KmsKeyArn value.
6. Click Validate. If a check fails, the page shows why. When the validation passes, click Save. A change to a value needs a new validation.

New metrics and traces go to your bucket within 30 minutes after the save.

The CloudFormation template is available at https://s3.us-west-2.amazonaws.com/oodle-configs/storage/s3/byob-v2.yaml. If you create the stack from the template directly, set the OodleRegion parameter to the region of your Oodle deployment.

Choose the bucket name

The stack always creates a new bucket. To choose its name, enter the name in the BucketName parameter of the stack. The name must not be in use in any AWS account. If you keep the parameter empty, AWS makes a unique name.

You cannot use a bucket that you already have. A bucket that the stack creates always has the settings that Oodle needs:

• The lifecycle rules below. Oodle writes temporary data to these prefixes. Without the rules, the temporary data stays in the bucket and your storage cost increases.
• Default encryption that agrees with the encryption that Oodle sends. Oodle sends the encryption in each write request, which overrides the bucket default. Without a KMS key on the settings page, Oodle writes with SSE-S3 (ServerSideEncryption: AES256), so a bucket policy that requires SSE-KMS rejects the writes.

If you delete the stack, AWS keeps the bucket and its data.

Lifecycle rules

The stack adds these rules to the bucket. The write-ahead log (WAL) prefixes of all signals use one expiry, which you can set with the WalExpirationDays parameter (default 3 days).

Prefix	Expire objects after
lambda_query_res/	1 day
wal/	WalExpirationDays
wal_late/	WalExpirationDays
wal_TRACE	WalExpirationDays
logs/wal	WalExpirationDays

A rule for the full bucket also deletes incomplete multipart uploads 1 day after they start.

Encrypt with a KMS key

S3 can encrypt each object with SSE-KMS and a customer managed KMS key in your account. SSE-KMS is optional. Without it, the bucket uses SSE-S3.

To use SSE-KMS, set Encryption to SSE-KMS on the settings page, then launch the stack. The link sets the stack parameters for you. The stack creates a KMS key with automatic rotation, and gives the Oodle role access to it. When the stack is complete, enter its KmsKeyArn output on the settings page.

Oodle then sends ServerSideEncryption: aws:kms, the key, and the S3 Bucket Key with each write, and S3 encrypts the object with the key. With this setting, a bucket policy that requires SSE-KMS accepts the writes.

What you must know about the key

• Do not disable or delete the key. Oodle cannot read or write the data in the bucket without it. When the key is not available, ingestion and queries fail. After the key is deleted, the data cannot be recovered.
• The KMS request quota of your account applies. S3 calls KMS for the role, which is in your account. The S3 Bucket Key keeps the number of KMS calls low. Do not turn it off.
• The S3 Bucket Key changes the encryption context. KMS calls show the bucket ARN, not the object ARN, in AWS CloudTrail and in the kms:EncryptionContext:aws:s3:arn condition key.

To use SSE-KMS with a bucket that you set up with the version 1 template, update the stack to the current template with Encryption set to SSE-KMS. Then enter the key ARN on the settings page, then Validate and Save. The objects that Oodle wrote before stay encrypted with SSE-S3.

IAM permissions

The trust policy of the role lets the Oodle AWS account assume the role only with your external ID. The role has these permissions, and only on your bucket:

Action	Resource
s3:GetObject, s3:PutObject, s3:DeleteObject, s3:AbortMultipartUpload	arn:aws:s3:::<bucket>/*
s3:ListBucket, s3:GetBucketLocation	arn:aws:s3:::<bucket>
kms:GenerateDataKey, kms:Decrypt (SSE-KMS only, through S3 only)	The KMS key

Oodle accepts only roles whose name starts with OodleS3Integration-BYOB- at path /. Keep the role name that the stack creates.

Change the bucket or the role

The bucket cannot be changed once it is set. Check the bucket name before you save.

You can replace the role or the KMS key for the same bucket. Enter the same bucket name, the role ARN, and the key ARN on the page, then click Validate and Save.