Kubernetes Postgres Backups
Services running in kubernetes backed by a datastore also in kubernetes require backups just like everything else. This post describes one way to make automated backups for a PostgreSQL DB in kubernetes.
With a few modifications, the approach detailed here can be applied to other databases that are not Postgres.
Requirements
- Backups must be taken regularly, once a week is fine for this example
- Backups must be stored on a separate host, as part of a 3-2-1 strategy.
- A 3-2-1 backup strategy is having 3 copies of data, on at least two types of media, with at least one of those copies being offsite. For this post I’ll be using my NAS as a second copy. The NAS will have several backups at any given time. The offsite backup is out of scope for this post.
- Obviously no secrets should be stored in git
- Backups should be encrypted at rest
Out Of Scope
- N backups are kept at any given time. I have other existing automation that handles this.
- Creating kubernetes secrets used in the backup. This post assumes the secrets are either already created, or created as part of your deploy job.
The Plan
- Create a new image based on bitnami’s postgresql image containing a shell script to dump, encrypt, and compress the a database
- Create a kubernetes CronJob to periodically backup the data
- Manually test to verify data is backed up
The Backup Shell Script
This is a quick bash script I’ve put together for taking backups. The script assumes two environment variables have been set.
- PGPASSWORD - used to access PostgreSQL database.
- ENCRYPT_PASSWORD - used to encrypt the dump with a password. With minor alterations to the script a pubkey could be used instead of a password. I chose a password because laziness. I mean seriously, I skipped an entire openssl command here.
If you run this script on something that is not ephemeral, you’ll want to update the script to remove ${TMP_OUT}
since that’s an unencrypted copy of a supposedly important database.
|
|
The Image
The Dockerfile here is very simple.
|
|
The CronJob Definition
Below is a CronJob used with an exampled application named “important-application-here”.
Several small pieces are left as an exercise for the reader.
- The image is left as
image: <Where-You-Host-Your-Image>:latest
. You’ll want to update that with the location of the image generated from the Dockerfile above. - For my purposes the volume
important-application-here-postgres-backup-nfs-claim
is assumed to already exist. I’ve used NFS mounts to mount my DB backup share on my NAS. - Creating secrets will need to be done.
You may also want to update the schedule. Having many services run their backups at the same time could cause an issue, but really that’s a problem for another day.
|
|
Verification
Once the CronJob has been deployed, you’ll want to verify it works. No one wants to wait up to a week to see if their yaml skills are up to par, so create a manual run.
|
|
Verify that your backup shows up in the expected location, and that is has the expected contents! I mounted the backup NFS on /mnt/db_backups
|
|
A smart person would also verify that the service can be restored from backup. That process depends on the application being backed up. Don’t wait until you need the backup to verify your recovery process works. If you don’t verify your recovery process, you do not have a recovery process.