TMC Data Protection is a relatively new feature in TMC that allows you to backup and restore your Kubernetes clusters, namespaces or label-specific objects. It is based on Velero, is easy to configure and even allows you to set schedules for regular backups. I’ve used Velero on it’s own in the past against workloads in TKG clusters but this was my first time using it in a more automated fashion with TMC…I was thoroughly impressed and wanted to share the experience.
Table of Contents
Create a TMC credential and AWS CloudFormation stack to be used by TMC Data Protection
These steps assume that you already have an IAM account setup on AWS. You can find detailed instructions for creating one in my previous blog post, How to deploy a TKG cluster on AWS using Tanzu Mission Control.
In the TMC UI, navigate to Administration and click the Create Account Credential dropdown and then choose AWS data protection credential.
Enter a descriptive name for the new credential and then click the Generate Template button.
Click the Next button.
The template that was created will be used to create a new CloudFormation stack. Head back to the main AWS page and navigate to CoudFormation under Management & Governance. Click the Create stack dropdown at the top right and then choose With new resources (standard).
Under Specify template, select Upload a template file. Click the Choose file button and then select the .template file that was created in TMC earlier.
Click the Next button.
Enter a name for the new stack and click the Next button.
Click the Next button on the Configure stack options page as there is nothing we need to change here.
Ensure everything is correct on the Review page. Click the checkbox at the bottom to acknowledge the IAM resources message and then click the Create stack button.
You should see that the creation is in progress.
You can click the refresh button and see various tasks running and completing.
Click on the Stack info link. When the Status is CREATE_COMPLETE, the stack is ready.
Click on the Outputs link and copy the ARN value noted.
You can now head back to TMC and click the Next button in the AWS Configuration section.
Click the Create Credential button.
You should now see your credential listed on the Administration > Accounts page.
Enable Data Protection on your cluster
In the TMC UI, navigate to Clusters and click on your cluster. Click on the Enable Data Protection link at the bottom of the page.
Select the credential you just created and click the Enable button.
You’ll see that Data Protection is being enabled in the TMC UI.
If you navigate to Storage > S3 at AWS, you should see a new S3 bucket being created.
When Data Protection is fully enabled, you’ll see a Create Backup link in the Data Protection pane for your cluster.
With Data Protection enabled you’ll see that there is a new namespace named velero with a single pod in it.
kubectl --kubeconfig=kubeconfig-clittle-test-cluster.yml -n velero get po
NAME READY STATUS RESTARTS AGE
velero-88ff97748-85tq7 1/1 Running 0 38mAt this point, you should install the velero executable so that you can have better visibility into your backups and restores. It’s a single file and you can find simple instructions for installing it at https://velero.io/docs/v1.4/basic-install/#install-the-cli.
Backup a workload with Data Protection
If you have a workload deployed, you can click the Create Backup link to test out this functionality. I have a WordPress application deployed in this cluster so I’ll be using that for this exercise. I can see that the site is up and has some data via a post I created on it.
You have several options to choose from when deciding how you’ll backup your workload. You can backup the entire cluster, a specific namespace or resources based on labels. In my case, all resources associated with the WordPress application have the app=wordpress label applied so I’ll use that option. Click the Next button when you’ve configured what you’ll be backing up.
We’re going to do a single backup so we can leave the Schedule type set to “Now” and click the Next button. I’ll circle back to some of the schedule options later.
The default retention period is 30 days and you can change this to suit your needs. Click the Next button when you’re ready to proceed.
Give your backup a name and click the Create button. You’ll be redirected to the Data Protection page for your cluster where you can watch the progress of the backup.
When the backup is finished you can click on it to see more details about what was backed up.
You can also drill down into the S3 bucket that was created at AWS to see contents of the backup.
You can also use the velero executable to see the backup and its details.
velero --kubeconfig=kubeconfig-clittle-test-cluster.yml get backups
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
wordpress-dp-backup Completed 2020-08-07 14:34:56 -0600 MDT 29d cjlittle-dp app=wordpressvelero --kubeconfig=kubeconfig-clittle-test-cluster.yml describe backup wordpress-dp-backup
Name: wordpress-dp-backup
Namespace: velero
Labels: tmc.cloud.vmware.com/creator=clittle
tmc.cloud.vmware.com/managed=true
velero.io/storage-location=cjlittle-dp
Annotations: velero.io/source-cluster-k8s-gitversion=v1.18.5+vmware.1
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=18
Phase: Completed
Namespaces:
Included: *
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: app=wordpress
Storage Location: cjlittle-dp
Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1
Started: 2020-08-07 14:34:56 -0600 MDT
Completed: 2020-08-07 14:35:02 -0600 MDT
Expiration: 2020-09-06 14:34:56 -0600 MDT
Persistent Volumes: 2 of 2 snapshots completed successfully (specify --details for more information)Restore the contents of a backup using Data Protection
I deleted my WordPress application after creating the backup.
kubectl --kubeconfig=kubeconfig-clittle-test-cluster.yml delete -k wordpress-aws/
storageclass.storage.k8s.io "k8s-policy" deleted
secret "mysql-pass-bd45fkk6kd" deleted
service "wordpress-mysql" deleted
service "wordpress" deleted
deployment.apps "wordpress-mysql" deleted
deployment.apps "wordpress" deleted
persistentvolumeclaim "mysql-pv-claim" deleted
persistentvolumeclaim "wp-pv-claim" deletedkubectl --kubeconfig=kubeconfig-clittle-test-cluster.yml delete pv --all
persistentvolume "pvc-3a3eef27-e8fa-497c-9491-3bda00225be1" deleted
persistentvolume "pvc-4a476072-a8aa-403b-88eb-650e3b044054" deletedkubectl --kubeconfig=kubeconfig-clittle-test-cluster.yml get deployment,replicaset,po,svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h15mAnd if I try to access my WordPress site, it’s not there anymore.
Let’s hope that TMC Data Protection can get it back!
When you’re ready to perform a restore, you’ll need to navigate to Clusters, select your cluster and then select the Data Protection tab. From here you will click on the backup (in the Backups section) that you want to restore.
On the details page for your backup, click on the Restore Backup button.
You can choose to either restore the entire backup or a selection from the backup based on included namespaces or labels. For my example, there should be no difference between selecting the entire backup or using the label selection that was used when creating the backup (app=wordpress). Click the Next button when you’re ready to move on.
Enter a name for your backup and click the Restore button.
You can follow the progress of the restore in the Restores pane of the Data Protection tab.
You can also use the velero CLI to get more information about the restore.
velero --kubeconfig=kubeconfig-clittle-test-cluster.yml get restores
NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR
wordpress-dp-restore wordpress-dp-backup InProgress 0 0 2020-08-07 14:56:31 -0600 MDT app=wordpressvelero --kubeconfig=kubeconfig-clittle-test-cluster.yml describe restore wordpress-dp-restore
Name: wordpress-dp-restore
Namespace: velero
Labels: tmc.cloud.vmware.com/creator=clittle
tmc.cloud.vmware.com/managed=true
Annotations: <none>
Phase: InProgress
Backup: wordpress-dp-backup
Namespaces:
Included: *
Excluded: vmware-system-tmc
Resources:
Included: *
Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
Cluster-scoped: included
Namespace mappings: <none>
Label selector: app=wordpress
Restore PVs: autoWhen the restore is done you’ll see the status as Ready on the Data Protection page.
And you should see all objects restored in your cluster.
kubectl --kubeconfig=kubeconfig-clittle-test-cluster.yml get deployment,replicaset,po,svc
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/wordpress 1/1 1 1 90s
deployment.apps/wordpress-mysql 1/1 1 1 90s
NAME DESIRED CURRENT READY AGE
replicaset.apps/wordpress-5fb76c9f88 1 1 1 90s
replicaset.apps/wordpress-mysql-5fcd84f896 1 1 1 90s
NAME READY STATUS RESTARTS AGE
pod/wordpress-5fb76c9f88-z7dvs 1/1 Running 0 90s
pod/wordpress-mysql-5fcd84f896-hv8h5 1/1 Running 0 90s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h52m
service/wordpress LoadBalancer 10.104.31.148 ab14057ea523e4721aeb075db9338685-1719023603.us-east-1.elb.amazonaws.com 80:30705/TCP 91s
service/wordpress-mysql ClusterIP None <none> 3306/TCP 91sThe external address for the wordpress service is changed since the Load Balancer was recreated at AWS but you can access the new address to validate that nothing was lost.
One thing worth mentioning if you’re going to try this exact scenario on your own. WordPress saves the loadbalancer URL internally so you will likely have some issues accessing the site after the restore. You can create a local host entry that maps the old address to the new loadbalancer IP address to get back in. Once in, you can update the URL that WordPress has saved to the new one.
As I mentioned earlier, you can schedule backups to occur on a recurring basis. The options are hourly, daily, weekly, monthly or on a custom schedule that you can specify in cron format.
In this example, I’m creating an hour backup that will run at 20 minutes past the hour and backup the entire cluster. As soon as the scheduled backup is created, a backup is taken and you will see the result of it in the Backups list. You’ll also see your new schedule in the Shedules list.
From the command line, using the velero command, we can see the schedule that has been created and some details about it.
velero --kubeconfig=kubeconfig-clittle-test-cluster.yml get schedules
NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR
hourly-cluster-backup Enabled 2020-08-28 09:15:49 -0600 MDT 20 * * * * 720h0m0s 11m ago <none>velero --kubeconfig=kubeconfig-clittle-test-cluster.yml schedule describe hourly-cluster-backup
Name: hourly-cluster-backup
Namespace: velero
Labels: tmc.cloud.vmware.com/creator=c_01EGTRHZ6NN94KB03RNHAPPZPE
tmc.cloud.vmware.com/managed=true
Annotations: <none>
Phase: Enabled
Schedule: 20 * * * *
Backup Template:
Namespaces:
Included: *
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: included
Label selector: <none>
Storage Location: cjlittle-dp
Snapshot PVs: false
TTL: 720h0m0s
Hooks: <none>
Last Backup: 2020-08-28 09:20:04 -0600 MDTI set the time for this recurrence such that I would get a scheduled backup to fire off a few minutes after I created is and we can see that there is a second cluster backup in the list at 20 minutes after the hour.
I decided to leave this running for a while to make sure it kept up with my schedule and so far nothing has been missed.
velero --kubeconfig=kubeconfig-clittle-test-cluster.yml get backups
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
hourly-cluster-backup-20200828232004 Completed 2020-08-28 17:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828222004 Completed 2020-08-28 16:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828212004 Completed 2020-08-28 15:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828202004 Completed 2020-08-28 14:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828192004 Completed 2020-08-28 13:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828182004 Completed 2020-08-28 12:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828172004 Completed 2020-08-28 11:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828162004 Completed 2020-08-28 10:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828152004 Completed 2020-08-28 09:20:04 -0600 MDT 29d cjlittle-dp <none>
hourly-cluster-backup-20200828151549 Completed 2020-08-28 09:15:49 -0600 MDT 29d cjlittle-dp <none>
wordpress-dp-backup Completed 2020-08-28 09:10:39 -0600 MDT 29d cjlittle-dp app=wordpressAnd if we take a look at our S3 bucket on AWS, we can see that there is a separate folder created for each backup.
Pingback: How to enable and use a custom S3 or S3-compatible storage location for Data Protection backups in TMC – Little Stuff