Cron schedule

[matusvlcek]

Hi everybody,
I have a problem with zrep in cron and do not know if it's a bug or just I'm doing something wrong. I'm syncing fs from master to slave server with command zrep -S containers/test. Everything works flawlessly. The problem is, when I put this into Cron. If I run it like this, it gives me this error and nothing happens. I do not have nested filesystems.

WARNING: old ZFS version detected with no depth protection
WARNING: You may not nest zrep managed filesystems

If I try to add sudo (even though cron runs as root), zrep makes a snapshot, but does not send it to slave server. It doesn't give an error, but only this line on output:

sending containers/test@zrep_00000b to vpn-server-2:containers/ahoj

In zfs list, I can see the snapshot, but marked as unsent.

containers/test@zrep_00000b_unsent     0B      -      128K  -

When I issue zrep -S containers/test, it gives me a nice output (debug is on) and sync with slave server.

DEBUG: zrep_lock_fs: set lock on containers/test
sending containers/test@zrep_00000d to vpn-server-2:containers/ahoj
Expiring zrep snaps on containers/test
DEBUG: expiring containers/test@zrep_000008_unsent
Also running expire on vpn-server-2:containers/ahoj now...
DEBUG: zrep_lock_fs: set lock on containers/ahoj
Expiring zrep snaps on containers/ahoj
DEBUG: expiring containers/ahoj@zrep_000008_unsent
zrep_unlock_fs: unset lock on containers/ahoj
zrep_unlock_fs: unset lock on containers/test

OS, SW: Ubuntu Server 20.04, kernel 5.4.0-91-generic, zfs-0.8.3-1ubuntu12.13, zrep 1.8.0

Do you have any clue, what could be wrong?

Thanks you very much for any info. I wasn't able to find any solution, what could solve this.

[ppbrown]

Seems like you are missing two things.
1 is details on how specifically you are running it. I would presume most people run it as just root to root.
If you are using sudo, then you should add details for that.

2 is of course, error messages :-}
If you would like to know more details of where things are breaking, start be explicitly running with the $DEBUG environment variable set to 1.
Beyond that, start dropping in "set -x" at the top; level, and then also to individual functions that get called.
Note that set -x output gets send to stderr. So any redirects you want, should be done with 2>/some/place/else.

[matusvlcek]

Thanks for the quick response,

I do everything as root user. Cron runs also as root. In file /etc/crontab, daily jobs (in my case) are set as root. I'm editing cron manually with command crontab -e and also with saltstack. This is the command in crontab:
0 2 * * * zrep -S containers/test 2>&1 > /var/log/zrep_cron.log

This is when it makes a ZFS error message mentioned above into zrep_cron.log
Then I just tried to use sudo in command inside crontab, not hoping anything would change, but I don't get any more error.
0 2 * * * sudo zrep -S containers/test 2>&1 > /var/log/zrep_cron.log

Instead of error, I just get this one line into zrep_cron.log. Servers do not sync, only snapshot on server 1 is created, but not send to server 2.
sending containers/test@zrep_00000b to vpn-server-2:containers/ahoj

Of course, $DEBUG variable in zrep is set to 1. But it looks like cron is ignoring this fact :).

I've just tried redirect only stderr 2> as you mentioned and apparently, my redirection stderr to stdout and to file 2>&1 > wasn't working as I wanted. Now I get:
root@vpn-server-2: Permission denied (publickey).
Error: Problem doing sync for containers/test@zrep_00000b. Renamed to containers/test@zrep_00000b_unsent

So, I suppose the problem is that cron cannot ssh to server 2. I'm gona take a look at that, I'll get you know if I find something.

[ppbrown]

yup figured it was somthing like that :)

remember also that usually, stuff in cron loses all environment setup. So you need to set env explicitly.
sometimes even $HOME.

[matusvlcek]

Thanks @ppbrown for your advices.

So the problem was SSH connection (in my case, password auth is disable, only publickey is allowed). Solved with generating ssh keys. Zrep was working fine in terminal, because zrep was forwarding ssh key from my session to second server, so I was able to connect without problem. I didn't realize this. Cron cannot use my ssh key, so it was rejected.
There was also second problem, the need of running zrep with sudo in cron with root privileges. This is almost certainly environment setup problem, because zrep run by cron was probably looking for zfs in /usr/bin, but zfs in my case is located in /usr/sbin and it puts error below. I fixed it with symlink.

stdout:
WARNING: old ZFS version detected with no depth protection
WARNING: You may not nest zrep managed filesystems
stderr:
/usr/bin/zrep: line 1959: zfs: not found
/usr/bin/zrep: line 1960: zfs: not found
Error: Problem getting zrep properties for fs containers

If someone needs help, this is the process I did:

1. Generate ssh key on server 1, issue a command ssh-keygen.
2. Copy your generated ssh public key from server 1 home directory .ssh/id_rsa.pub and append it to .ssh/authorized_keys on server 2
3. Do the same thing on server 2 and copy its public key to server 1.
4. Create symlink for zfs, issue command on both servers ln -s /usr/sbin/zfs /usr/bin/zfs