Track S3 Bucket Changes using Lambda

In this guide, We will learn how to Configure S3 buckets and Lambda function to track any changes that occurs in S3 buckets such as uploading files , removing files etc.

During such actions in the S3 buckets , It will trigger the Lambda function and the response will be logged in Cloudwatch Logs.

Services Involved

• S3 Bucket
• Cloudwatch
• Lambda

Steps Involved

• Create S3 bucket.
• Create Role and add permissions for the S3 buckets to work with Lambda functions.
• Create and configure Lambda function with S3 bucket as Triggers.

Once the user upload the file to S3 bucket , It will trigger the lambda function (as it has necessary execute function) , The lambda will then send the response to the Cloudwatch logs where the users can track the changes such as uploading files.

Creating S3 Buckets

I have explained in detail in my previous article on how to create and manage S3 buckets , Refer this article.

Also, Check the more article related to S3.

Setup S3 Bucket Cross region replication

Cross domain resource sharing in S3 bucket

Automate backup of files from Windows to S3 bucket

To create S3 buckets using AWS Console , Login to S3 Console

Choose Create bucket

Provide a unique name for the bucket and choose the AWS region where the bucket should be created.

and click Create bucket

Now that we have created S3 bucket.

Creating IAM Role

To create IAM Role , Login to IAM Console

Choose Roles and then click Create role

Under AWS Service , Choose Lambda

And click Next: permissions , here we have to grant full access permissions such as S3 , Lambda and Cloudwatch.

And then click Next:Tags

Provide a name for the role and then click Create role

Now we have created IAM role and attached the necessary permissions with it.

Create & Configure Lambda

To create Lambda function , Login to Lambda Console

Click Create function , Choose Author from scratch

For Basic information , Provide a name for the function

Choose Runtime as Node.js 12.x

For permissions , Under Choose or create an execution role , Select use an existing role

and click Create function.

Now that we have created a Lambda function.

Lets go ahead and configure the lambda function such as adding S3 buckets as triggers.

Under Function code

Replace the existing code with the code attached below.

exports.handler = function(event, context, callback) {
   console.log("Incoming Event: ", event);
   const bucket = event.Records[0].s3.bucket.name;
   const filename = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
   const message = `File is uploaded in - ${bucket} -> ${filename}`;
   console.log(message);
   callback(null, message);
};

and then click Save.

Under Designer , Click + Add trigger

For trigger configuration , Select a trigger as S3

Under Bucket , Select the bucket you have created.

Under Event type , Choose All object create events

Check Enable trigger and then click Add.

Now you will able to see the S3 triggers If you choose S3 under Designer.

Testing Lambda Trigger

Now We have to upload a file to the S3 bucket , To check , whether it triggers the lambda function and it outputs the response to Cloudwatch Logs.

Go to S3 console , Select the bucket you have created.

Choose Upload , Click Add files , It will take you to the file manager , Select the file and then click Upload.

To verify the same , Go to Cloudwatch Console , In the left navigation pane , Under Logs , Select Log groups

And search for /aws/lambda/functionname

Select the Log group and then select the Log stream

You will see the response as shown below.

INFO File is uploaded in - bucketname -> filename

This shows that the Whenever we upload a file to the s3 bucket , It triggers the lambda function and output the response in Cloudwatch console.

Conclusion

We have configured S3 buckets and Lambda function to track S3 bucket changes such as uploading files and log the response in the Cloudwatch log.

Hope you found it helpful.Thanks for reading this article.

Please do check out my other publications.