Backup ZFS snapshots to cloud storage using RClone rcat
This requires auto-zfs-snapshot to run at least daily and hourly backups and a working rclone remote, frequent snapshots are recommended. No snapshots are created, only holds placed and released
To create the initial backup, the yearly script sends the latest snapshot (including frequent or hourly snapshots) to a path in an rclone remote. This must be run manually to create the initial databackup and run as a cronjob on January 1st every year.
A cronjob for daily and monthly snapshots should be created passing the required parameters. The last snapshot with an rclone-backup hold will be used to send the incremental dataset to rclone. if no last snapshot exists with a hold, the most recent snapshot will be used. even if the last snapshot is not finished uploading, the next snapshot will continue to process.
The holds are only used to ensure the same reference snapshot is available for the next time this script runs. This allows you to manage how long a snapshot persists with other tools. The intention for data stored in an rclone remote is to keep 1 yearly initial snapshot, 11 monthly snapshots, and 90 daily snapshots in a rolling fashion.
Some additional benefits of using rclone is the ability to combine data transformations inline from the input data to the backing cloud storage. It is possible that a single initial snapshot can exceed the maximum file limit, such as 5TB in google drive, so the rclone chunker remote is used to split files into 4TB segments. Even if rclone becomes unsupported, these files can be manually downloaded and concatenated using linux tools. Furthermore, an rclone compress remote can be used to reduce the amount of data sent in bandwidth limited or metered internet connections. These files can be expanded using linux tools without rclone If file encryption is desired, a crypt remote should be placed between the cloud storage remote and the compress remote specifically because encrypted data can not be compressed. The encrypted foldernames and files can not be reversed without rclone, however in the case that rclone support for the cloud storage is depreciated, it can be used to read locally downloaded files and decrypt them.
Restoring
restoring requires every snapshot file from the rclone remote to be downloaded and piped through zfs receive from the oldest yearly to the most recent daily snapshot. a script to restore data is being considered once backups are consistent. piping the output or rclone ls remote:path/to/snapshots through xargs to execute "rclone cat $1 | zfs receive recoverpool/recoverdata" is a manual method for this restoral
Any pull requests appreciated.