Backup et restore persistent volume

[jraboeuf]

Hello,

I begin with vDVS and I would like to know how to backup and restore properly a persistent docker volume.
I already tried snapshot or Veeam Backup but a restore doesn't work
Have you a good methode ?

Thx in advance

Jeremie

[govint]

@jraboeuf, thanks for trying out the the vDVS plugin could you describe the use case you are attempting with the Veeam backup. Can you also say how you configured Veeam to take a snapshot. Is it for a specific VMDK or set of VMDKs for the containers on a docker host. vDVS volumes are independent volumes and hence not sure that Veeam will snapshot and backup those.

[jraboeuf]

My example :
1 postgresql container on Docker swarm cluster (on VM)
1 Docker Volume "persistent" on specific datastore

When the container run, it's running on 1 of host on the swarm cluster, this host have the Docker volume mapped and I can backup it with Veeam but if I need to restore it, I can't do it.

I already tried this restore methodes :

• Restore virtual disk but docker host is stopped during the operation and the volume not recognize properly after reboot.
• Restore VM Files but I have the same issue
• Restore guest files, it possible to restore some files but not with the DB files

I don't have a specific configuration on Veeam but I have select this options :

Have you an idea ?
Jeremie

[ghost]

May I add my two cents to the question. There is a need for backing up persistent volumes as there is a need to backup any type of data. The data may have been deleted by accident or the user may want to restore the data as it was at a specific date. How do we do that ?
Any recommended approach ?

[AndrewSav]

@govint it is not clear what's vSphere support for backup of independent disks. So far I could find none. Veem does not support it. I cannot really use vsphere-storage-for-docker if I cannot do regular backups of it. How are we supposed to backup these? What's the proposed "official" solution?

One wants to use vsphere-storage-for-docker in a docker cluster, so that it's available regardless what node in the cluster an app is scheduled to. In this respect vsphere-storage-for-docker performs admirably. But since these volumes is the only state the apps have it's extremely valuable and needs to be backed up.

[carcampbell]

echoing the statements above. We want to build an enterprise solution and need backup for the persistent volumes. The point of having persistent data is so it's always there and useable but if not backed up (with of course ability to restore!) consistently there is great potential for loss of data and outages/customer dissatisfaction. does VMware have a solution or recommendation?

[sege]

I'm with the above. I'm really eager to try this out for real but without backup and restore I have a hard time using this for real.

Any ideas? From VMware? From the community?

[jraboeuf]

My only way to Backup/Restore files in persistent volume with vDVS is give a constraints to containers on 1 host and only possible to restore simple files not DB :(

[patrikwm]

This is just what I have been thinking about last couple of days. Would be great to have some backup strategy.

[govint]

Presently, the plugin only implements the docker plugin interface and can create and attach these volumes only to containers. The vSphere features for backup, supported for VMDKs, may not apply for these volumes (at least not wholly). There is no way presently, to make snaps of individual container volumes (snapshotting the whole VM seems just wrong if multiple volumes are in use and don't need to be snap'ed).
Backup apps shouldn't expect to backup container volumes the way VMDKs are. Yes, they are VMDKs on the backend but these are container volumes and the way to access is to have them attached to a container and then backup data. The sequence should be like,

1. Stop app.
2. Attach container volume to backup container.
3. Backup data and terminate backup container.
4. Start app.
   That's possibly what will work for now. I'm not sure the volume plugin should attempt to build out backup and restore within itself or as extensions. The user should be able to build this capability around what the volume plugin provides as a volume provisioning entity.

[AndrewSav]

@govint One can back up a VM without downtime. Stopping the app sounds severe. If I do daily backups the app will have to be down for a few minutes a day, which is especially problematic with databases. I understand that there is not better solution right now and I thank you for your thoughts.

[govint]

Thinking this thru a bit more, possible approaches could be as below.

Backup volume data on unmount:

1. When a container stops running, the volume is unmounted, the plugin watches for the last unmount (in the case that the volume is used by multiple containers on the host) and proceeds to create a backup of the volume. The exact implementation of the backup could be a clone of the volume or a compressed image of the disk contents stored separately and locally (later to more storage backends as thats feasible).
   a. The plugin backs up all data in the volume while the app may not necessarily be wanting everything backed up. See below how this may be improved.
   b. The plugin supports new options on volume create to support backup and TTL for backups.
   i. enable - enable backup of the volume. Volume inspect on the volume lists the backups available.
   ii. ttl - time to live for the backup, the plugin will delete backups older than this time for backups of the volume.
   The plugin adds meta-data per volume to track what backups are available and the time at which they were created.
   c. To restore a backup the user inspects the volume to figure what backups exist for it.
   d. Do volume create to create a new volume with options specifying a) the source volume and b) the specific backup of that volume. The driver creates the new volume and populates it with the data from the specified backup.
   e. Deleteing a volume removes all of its backups as well. Except those volumes that were explicitly created from backups (d) above.

To reduce the amount of backed up data to exactly what the application wants, the user provides volume create time options to specify a) a backup command and b) a restore command that is run by the volume plugin when it creates a backup or restores from a backup.
The command gets a source and destination file paths that the command can then use to pick specific data off the source and write to the destination, to create the backup . The user gets to provide these commands and hence control exactly what data to backup and restore. Rather than have the plugin create backups that are full copies of the application data.

This approach can at best restore to the point when the volume was last unmounted (or the application stopped running). These aren't incremental either as there is no way to figure what blocks changed since the last backup anyway. Does this work better than no backups?

[ghost]

So the apps would need to unmount a volume to trigger a backup ?

Mmm not sure it helps. Can we expect the following to be fixed ? It would also help I believe
#2065

[AndrewSav]

@chris7444 can you fix the link please?

[ghost]

issue 2065

#2065

[govint]

@chris7444, I don't believe we can take a clone of a volume when its still in use. Then again, how is app data consistency handled if volume data is copied over arbitrarily.

[ghost]

@govint It is up to the application to decide this. Many applications have what it takes to recover data from "crash consistent" backups. Who is stopping a DB today or unmounting the FS that contains the data to take a backup or snapshot ?

[jraboeuf]

Yes it is axectly my use case, how to backup DB with no stop container who attach volumes ?

[govint]

OK, verified that the clone of a volume won't happen unless the source volume is not in use. We can do the clone of a container volume provided all containers that are presently using the volume are paused, the volume is cloned and then containers are resumed. Which should allow a PIT copy of the container volume to be created. Which could actually be scripted vs. needing any code to handle this in the plugin.

The container volumes are configured as independent disks to the VM and as such hot cloning isn't possible and needs the volume to not be in use.

[ghost]

@govint Let me make sure I understand

user wants to clone volume X

1. user pauses all containers accessing volume X
   docker pause...

2. user clones the volume
   docker volume create --driver=vsphere --name=CloneVolumeX -o clone-from=VolumeX -o access=read-only

3. user resumes the containers
   # docker unpause...

is that it ?

[govint]

My bad, I should have verified the flow first. This will need some changes on the ESX host side to handle the clone. Let me make those changes, try it out first.

[govint]

I'll post a change to allow taking a hot clone of a container volume, user should be able to pause container - clone volume - resume container. Cloned volume would be a PIT backup of the container volume. With the fixes in #2077 merged this should be doable now.