Implement RDS snapshot sanitizer #1
Conversation
| - `SANITIZER_DELETE_OLD_SNAPSHOTS`: (Optional) Whether to delete old snapshots. Defaults to False. | ||
| - `SANITIZER_OLD_SNAPSHOT_DAYS`: (Optional) Number of days for a snapshot to be considered old. Defaults to 30. | ||
|
|
||
| # Configuration |
There was a problem hiding this comment.
Could you add a sample of the configuration?
|
|
||
| Create sanitized copy of RDS snapshots and share them with selected accounts. | ||
|
|
||
| It works by restoring an unsanitized snapshot to a temporary cluster and executing sanitizing SQL queries against it, after which sanitized snapshot will be created and optionally shared with other accounts. |
There was a problem hiding this comment.
I think it's better if you provide guide on how to run this locally (either with Docker or with Poetry)
There was a problem hiding this comment.
Also, I think it's useful to describe what IAM permissions are required for this
There was a problem hiding this comment.
Also, I think it's useful to describe what IAM permissions are required for this
Yes, I'm thinking of creating a pod-identity terraform module for it in the style of https://github.com/terraform-aws-modules/terraform-aws-eks-pod-identity
I think it's better if you provide guide on how to run this locally (either with Docker or with Poetry)
The thing is the tool needs to be connected to the RDS subnet for running the SQL query. I'll probably add a flag to set the postgres host to localhost, with the assumption that the user can connect their localhost to the RDS (via bastion port-forwarding for example).
There was a problem hiding this comment.
I see, if it's expected to run within AWS it's fine too, just add a note describing as such.
| click.echo("") | ||
|
|
||
| click.echo("#################### Creating shared snapshot ####################") | ||
| shared_snapshot = share_snapshot(sanitized_snapshot) |
There was a problem hiding this comment.
The shared snapshot is always created even if SANITIZER_SHARE_ACCOUNT_IDS is not set?
There was a problem hiding this comment.
Right, will make the step optional 🙏
| DBClusterIdentifier=restored_cluster["DBClusterIdentifier"], | ||
| AutoMinorVersionUpgrade=False, | ||
| BackupRetentionPeriod=1, | ||
| PreferredBackupWindow="22:00-22:30", |
There was a problem hiding this comment.
If not set, there is a possibility that RDS will execute automatic backup just after the temporary cluster is created, prolonging the wait time (this happened to me once).
Unfortunately RDS aurora does not support disabling automatic backup, so I set it to a time outside the usual backup window.
This implements a tool to sanitize RDS snapshots for sharing with external accounts.
Note that this is a public repository.