Syncoid pruning seems to be applied to target

[mc2contributor]

I thought I had everything configured correctly to keep long-term backups on the target server, but found out that both the source and target are pruning on the same schedule. It looks like syncoid is replicating pruning, though I don't believe that's how syncoid actually works. Based on #923 (comment), my sanoid.conf on the target is correct:

[pool/filesrv/home]
	use_template = zfs
	yearly = 5

...

# templates
[template_zfs]
	frequently = 0
	hourly = 30
	daily = 90
	monthly = 12
	yearly = 2
	autosnap = no
	autoprune = yes

	# hourly in minutes; daily in hours
	hourly_warn = 2880
	hourly_crit = 3600
	daily_warn = 48
	daily_crit = 60

Why would my backups on the target be pruned at 3 months, which is how sanoid is configured on the source, but not on the target?

[phreaker0]

syncoid can destroy snapshots on the target with the "--delete-target-snapshots" option.
if you didn't instruct syncoid to do it then you should run sanoid with the "--debug" flag, it will show you the effictive policy for your dataset.

[mc2contributor]

Thanks! syncoid shouldn't be the problem, as I'm using the following command to pull from the source:

syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --compress=zstd-fast zfsbackup@filesrv:pool/home pool/filesrv/home

Sanoid config seems fine too:

****** CONFIGS ******
$VAR1 = {
          'pool/filesrv/home' => {
                                   'autoprune' => 1,
                                   'autosnap' => 0,
                                   'capacity_crit' => '95',
                                   'capacity_warn' => '80',
                                   'daily' => '90',
                                   'daily_crit' => '60',
                                   'daily_hour' => '23',
                                   'daily_min' => '59',
                                   'daily_warn' => '48',
                                   'force_post_snapshot_script' => '',
                                   'frequent_period' => '15',
                                   'frequently' => '0',
                                   'frequently_crit' => '0',
                                   'frequently_warn' => '0',
                                   'hourly' => '30',
                                   'hourly_crit' => '3600',
                                   'hourly_min' => '0',
                                   'hourly_warn' => '2880',
                                   'monitor' => 1,
                                   'monitor_dont_crit' => 0,
                                   'monitor_dont_warn' => 0,
                                   'monthly' => '12',
                                   'monthly_crit' => '40d',
                                   'monthly_hour' => '0',
                                   'monthly_mday' => '1',
                                   'monthly_min' => '0',
                                   'monthly_warn' => '32d',
                                   'no_inconsistent_snapshot' => '',
                                   'path' => 'pool/filesrv/home',
                                   'post_snapshot_script' => '',
                                   'pre_pruning_script' => '',
                                   'pre_snapshot_script' => '',
                                   'prune_defer' => '0',
                                   'pruning_script' => '',
                                   'script_timeout' => '5',
                                   'skip_children' => '',
                                   'weekly' => '0',
                                   'weekly_crit' => '0',
                                   'weekly_hour' => '23',
                                   'weekly_min' => '30',
                                   'weekly_warn' => '0',
                                   'weekly_wday' => '1',
                                   'yearly' => '5',
                                   'yearly_crit' => '0',
                                   'yearly_hour' => '0',
                                   'yearly_mday' => '1',
                                   'yearly_min' => '0',
                                   'yearly_mon' => '1',
                                   'yearly_warn' => '0'
                                 },
        ...
        };

Filesystem pool/filesrv/home has:
     63 total snapshots (newest: 1.0 hours old)
          3 monthly
              desired: 12
              newest: 71.0 hours old, named autosnap_2024-06-01_00:00:02_monthly
          30 daily
              desired: 90
              newest: 23.0 hours old, named autosnap_2024-06-03_00:00:02_daily
          30 hourly
              desired: 30
              newest: 1.0 hours old, named autosnap_2024-06-03_22:00:02_hourly

but I just lost my March monthly. Anything else I can look into? I'm running version 2.2.0.

[phreaker0]

then you should look into you logs for the sandoid-prune.service if you are using it, if it was sanoid who removed the snapshots.
You should also check "zpool history" for the prune command and look at the timestamp, maybe you can connect sanoid or syncoid schedule times to it.

[mc2contributor]

I checked the logs and didn't find any record of pruned monthlies on the target, either with "zpool history" or "journalctl -u sanoid-prune", so I decided to stop sanoid on the target. Snapshots continued to disappear, which confirmed an issue in syncoid.

So I ran syncoid with '--debug', and found that it runs 'zfs send -R ... | zfs receive -F ...'. The man page for 'zfs-send' says the following about the '-R' option:

If the -F flag is specified when this stream is received, snapshots and file systems that do not exist on the sending side are destroyed.

I didn't set '--delete-target-snapshots', so this looks like a bug.

[phreaker0]

There is no bug. --delete-target-snapshots is a syncoid feature where it actively looks at the snapshots on the source and target an will remove the ones on the target which are not on the sending side.

But you are doing it directly with zfs with the -R sendoption.

[mc2contributor]

Solved by running syncoid with --no-rollback. This removes the '-F' option from 'zfs receive'.

It should also work to remove the --sendoptions=R option from syncoid. I haven't tried this, but it seems valid based on the 'zfs-send' documentation on the '-R' option. I don't know what effect this would have on my existing sync setup, so I've left that as-is.

[phreaker0]

IMHO --no-rollback should always be used if the data/snapshots on the target aren't modified by the target directly. Because if it fails because a rollback is needed in this scenario it won't lead to potential data loss.