README.md

S3 Changelog

For large buckets, scanning all objects might take a lot of time. This CloudFormation template enables the Changelog feature for S3 buckets, which improves the scan performance after the initial scan of the entire bucket is done.

This template deploys an SQS queue and an EventBridge rule which sends S3 update events to the queue. Then, the Elastio iscan job reads those events to perform the scanning of new objects.

Deploying the CFN stack

1. First, you need to enable Amazon EventBridge for your S3 buckets by following these instructions: Enabling Amazon EventBridge.

2. Use one of the following quick-create links. Choose the region where your Elastio Cloud Connector is deployed.

   Important! You can change the stack name, but it MUST start with elastio-. Otherwise, Elastio won't be able to access the created resources.

   • us-east-1
   • us-east-2
   • us-west-1
   • us-west-2
   • eu-central-1
   • eu-west-1
   • eu-west-2
   • eu-west-3
   • ca-central-1
   • ap-south-1
   • ap-southeast-1
   • ap-southeast-2
   • me-central-1

3. Fill in the main parameters:

   • BucketNames - comma-separated list of S3 bucket names;

   • ScanExistingObjects - if set to true, the full initial scan of the bucket will be performed. If set to false, only new objects in the bucket will be scanned. This option can only be enabled in non-cross-account scenario. If it is not set to false in the cross-account case, an error about missing AWS Lambda functions will occur during stack creation.

   • KeyPrefixes - (optional) comma-separated list of prefixes of objects to scan. This will be applied to all buckets. If you want to use different prefixes for different buckets, you need to deploy multiple S3 Changelog stacks. Also, note that the paths selector in the Protection Policy will also be used to filter objects before scanning. This means that the KeyPrefixes parameter must be in sync with the paths selector in the Protection Policy, or not specified at all;

   • DisableEventCollection - if you decide to disable the protection policy for your S3 bucket, you can set this to true to stop accumulating S3 update events in the SQS queue. IMPORTANT: If you set this parameter to true, then no more objects will be scanned on this bucket. If you set it back to false, then any new objects after the parameter is set to false will be available to scan, but all changes made to the bucket during the time when this parameter was set to true will not be scanned, unless you manually initiate a full scan of the entire bucket.

   There are other advanced and experimental parameters in the template, you can ignore them.

4. Check the box in front of I acknowledge that AWS CloudFormation might create IAM resources with custom names and I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND and click Create stack.

5. Create a protection policy for your buckets in Elastio Tenant.