A simple Lambda function for archiving RDS logs to a S3 bucket.
See the blog post for more details.
Since the release of this Lambda function, AWS RDS has added native support for shipping some logs to Cloudwatch Logs, which you can then ship to S3 too.
This Lambda function is therefore only useful if you're using a database/version that doesn't support log shipping to Cloudwatch, want to ship unsupported logs, or if you want to bypass Cloudwatch for some reason, e.g. cost.
The Lambda function can be executed locally by running it with some command-line arguments.
You will need to have a Python environment with boto3 installed, e.g.
python3 -m venv .venv
. .venv/bin/activate
pip install boto3(This was last tested with the python3.8 lambda runtime, using boto3==1.16.31 botocore==1.19.31. Substitute that in for boto3 in the pip command above to install these specific versions.)
Then you can run the function and see the necessary command-line arguments,
python rds_logs_archiver.py(It isn't necessary to include boto3 when running as a Lambda function as it is available as part of that environment.)
The easiest way to deploy this is to use the CloudFormation template.
If you want to deploy using other mechanisms, e.g. Terraform, it's best to look at the CloudFormation and translate it from there. It contains all the configuration and security permissions necessary.
To deploy using the CloudFormation template,
-
Create the S3 bucket to archive the log files to. You may want to add a retention policy to purge logs after a period of time.
-
Create a ZIP file with
rds_logs_archiver.pyin it. It must be namedrds_logs_archiver.pyand must live in the root of the ZIP file. If it isn't, Lambda won't be able to find the code and run it.zip lambda_code.zip rds_logs_archiver.pyshould do the trick. -
Upload the ZIP file to an S3 bucket so Lambda can use it. It can be the same as the bucket the log files will be archived to.
-
Create the stack in CloudFormation using the template. Change the parameters as necessary to reflect your setup, e.g. database identifier and bucket to archive logs to.
The Lambda function should execute within the next 10 minutes, and every 10 minutes after that.
If you want, you can manually trigger an execution by navigating to the function in the AWS console, and sending a test event with an empty payload (i.e. {}).