No snapshots no sync. Lost days of data transfer again?

[slrslr]

Hello, after sync:
syncoid -r --skip-parent pool poolbackup
it worked to sync and "USEDSNAP" column of "zfs list -o space" command output shown significant used space by snapshots. So i wanted to free up that:
zfs list -H -o name -t snapshot | xargs -n1 sudo zfs destroy
But second sync. run ends with CRITICAL ERROR: Target poolbackup/dataset exists but has no snapshots matching with pool/dataset! Replication to target would require destroying existing target. Cowardly refusing to destroy your existing target.
So it seems like that if i want to keep that backup pool in sync (beside using tools like rsync) using ZFS tools like syncoid, I need to spend another week copying all that data again right? And once i waste that time and electricity, after initial sync is done, what should i do to keep minimum disk space utilized by snapshots and at same time still being able to sync? I prefer to solve this using syncoid command instead of creating config. files. This is cold backup drive ran from time to time.
Thanks a lot in advance for possible help.

[phreaker0]

You need to read up on the basics of ZFS in order to use it properly. If you destroy all snapshots, there is no (common) snapshot any more to replicate.

[redmop]

Syncoid syncs from previous snapshot to current snapshot. If you delete all the snapshots on the destination, it won't have a previous snapshot to sync from. Basically, don't delete @syncoid_ snapshots, and consider --no-stream.

Yes, you probably need to start over. Don't delete the @syncoid_ snapshots if you use that command line. If you use --no-sync-snap, don't delete the latest snapshot.

[slrslr]

I was checking yours described parameters at https://github.com/jimsalterjrs/sanoid#sanoid-command-line-options
and this is what i have now:
syncoid -r --skip-parent --no-stream pool poolbackup
But that does not cause maximum disk space savings by removing older than recent snapshot, which command or command parameters i am missing to ensure only latest snapshot?

[phreaker0]

yes it does, this is the correct command. The target will only have the latest common snapshot. See the following example:

+ POOL_IMAGE=/tmp/syncoid-test-0.zpool
+ POOL_SIZE=1000M
+ POOL_NAME=syncoid-test-0
+ truncate -s 1000M /tmp/syncoid-test-0.zpool
+ zpool create -m /tmp/lala1 -f syncoid-test-0 /tmp/syncoid-test-0.zpool
+ trap cleanUp EXIT
+ zfs create syncoid-test-0/src
+ zfs create syncoid-test-0/src/1
+ zfs create syncoid-test-0/src/2
+ zfs create syncoid-test-0/dst
+ syncoid -r --skip-parent --no-stream syncoid-test-0/src syncoid-test-0/dst
INFO: --no-stream selected; sending newest full snapshot syncoid-test-0/src/1@syncoid_stan_2022-10-19:09:43:47-GMT02:00 (~ 12 KB) to new target filesystem:
44,2KiB 0:00:00 [8,41MiB/s] [================================================================================================================================================================================================================================] 350%            
INFO: --no-stream selected; sending newest full snapshot syncoid-test-0/src/2@syncoid_stan_2022-10-19:09:43:47-GMT02:00 (~ 12 KB) to new target filesystem:
44,2KiB 0:00:00 [9,58MiB/s] [================================================================================================================================================================================================================================] 350%            
+ zfs list -t snapshot -r syncoid-test-0
NAME                                                             USED  AVAIL     REFER  MOUNTPOINT
syncoid-test-0/dst/1@syncoid_stan_2022-10-19:09:43:47-GMT02:00     0B      -       24K  -
syncoid-test-0/dst/2@syncoid_stan_2022-10-19:09:43:47-GMT02:00     0B      -       24K  -
syncoid-test-0/src/1@syncoid_stan_2022-10-19:09:43:47-GMT02:00     0B      -       24K  -
syncoid-test-0/src/2@syncoid_stan_2022-10-19:09:43:47-GMT02:00     0B      -       24K  -
+ sleep 1
+ syncoid -r --skip-parent --no-stream syncoid-test-0/src syncoid-test-0/dst
Sending incremental syncoid-test-0/src/1@syncoid_stan_2022-10-19:09:43:47-GMT02:00 ... syncoid_stan_2022-10-19:09:43:49-GMT02:00 (~ 4 KB):
 624 B 0:00:00 [ 309KiB/s] [================================>                                                                                                                                                                                                 ] 15%            
Sending incremental syncoid-test-0/src/2@syncoid_stan_2022-10-19:09:43:47-GMT02:00 ... syncoid_stan_2022-10-19:09:43:49-GMT02:00 (~ 4 KB):
 624 B 0:00:00 [ 260KiB/s] [================================>                                                                                                                                                                                                 ] 15%            
+ zfs list -t snapshot -r syncoid-test-0
NAME                                                             USED  AVAIL     REFER  MOUNTPOINT
syncoid-test-0/dst/1@syncoid_stan_2022-10-19:09:43:49-GMT02:00     0B      -       24K  -
syncoid-test-0/dst/2@syncoid_stan_2022-10-19:09:43:49-GMT02:00     0B      -       24K  -
syncoid-test-0/src/1@syncoid_stan_2022-10-19:09:43:49-GMT02:00     0B      -       24K  -
syncoid-test-0/src/2@syncoid_stan_2022-10-19:09:43:49-GMT02:00     0B      -       24K  -
+ sleep 1
+ syncoid -r --skip-parent --no-stream syncoid-test-0/src syncoid-test-0/dst
Sending incremental syncoid-test-0/src/1@syncoid_stan_2022-10-19:09:43:49-GMT02:00 ... syncoid_stan_2022-10-19:09:43:50-GMT02:00 (~ 4 KB):
 624 B 0:00:00 [ 204KiB/s] [================================>                                                                                                                                                                                                 ] 15%            
Sending incremental syncoid-test-0/src/2@syncoid_stan_2022-10-19:09:43:49-GMT02:00 ... syncoid_stan_2022-10-19:09:43:50-GMT02:00 (~ 4 KB):
 624 B 0:00:00 [ 290KiB/s] [================================>                                                                                                                                                                                                 ] 15%            
+ zfs list -t snapshot -r syncoid-test-0
NAME                                                             USED  AVAIL     REFER  MOUNTPOINT
syncoid-test-0/dst/1@syncoid_stan_2022-10-19:09:43:50-GMT02:00     0B      -       24K  -
syncoid-test-0/dst/2@syncoid_stan_2022-10-19:09:43:50-GMT02:00     0B      -       24K  -
syncoid-test-0/src/1@syncoid_stan_2022-10-19:09:43:50-GMT02:00     0B      -       24K  -
syncoid-test-0/src/2@syncoid_stan_2022-10-19:09:43:50-GMT02:00     0B      -       24K  -
+ cleanUp
+ zpool export syncoid-test-0

[slrslr]

correct command

Thank you. I suspect that maybe the downside of the --no-stream is that when you are doing very long initial sync and the process interrupts, you can not resume it, but i may be wrong and following is not related.

Resuming interrupted zfs send/receive from pool/data to poolbackup/pool/data (~ UNKNOWN remaining):
cannot resume send: 'pool/data@syncoid_abc_2022-10-19:14:02:45' used in the initial send no longer exists
cannot receive: failed to read from stream
WARN: resetting partially receive state because the snapshot source no longer exists
INFO: --no-stream selected; sending newest full snapshot

Btw. i suggest to delete datasets at destination or re-create pool on destination to ensure faster speed.

[jimsalterjrs]

Syncoid manages its own synchronization snapshots. It will never delete any snapshot you took manually, but it will remove no-longer-needed sync snapshots (identifiable by the name "syncoid" in them) as they become unnecessary, that's half the point of them.

Unrelatedly, this won't work:

syncoid -r --skip-parent --no-stream pool poolbackup

because you can't replicate to a pool's root dataset. This WOULD work:

syncoid -r --skip-parent --no-stream pool poolbackup/pool

... although you would not be able to replicate in reverse to restore, since as noted above, you can't use a pool's root dataset as a replication target.

[slrslr]

Thanks for Your attempt to explain.
I do not do any manual snapshots since i do not understand it and may not need it.
My only aim is to sync two drive pool:

        NAME                                           STATE     READ WRITE CKSUM
        pool                                           ONLINE       0     0     0
          ata-Samsung_SSD_REDACTED  ONLINE       0     0     0
          ata-Samsung_SSD_REDACTED  ONLINE       0     0     0

(has no data in its root, only has data in its 2 datasets)

and sync it to a single drive backuppool to ideally end up with 1:1 clone. That is why i would run "syncoid -r --skip-parent --no-stream pool poolbackup" and and assume it will just sync datasets from pool to backuppool, but no, you said that it would not work (i do not know why while "syncoid -r --skip-parent pool poolbackup" worked when it had snapshots) and it is bad, because for simplicity it should just:

syncoid sourcepool destinationpool