Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

syslog-ng constantly crashing #6118

Open
5 tasks done
CouponCodeSwap opened this issue Oct 19, 2024 · 16 comments
Open
5 tasks done

syslog-ng constantly crashing #6118

CouponCodeSwap opened this issue Oct 19, 2024 · 16 comments
Labels
stale Please update the issue with current status, unclear if it's still open/needed. unconfirmed

Comments

@CouponCodeSwap
Copy link

Contribution guidelines

I've found a bug and checked that ...

  • ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
  • ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
  • ... I have understood that answers are voluntary and community-driven, and not commercial support.
  • ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

This is a report in regards to the following issue reported by @chriscroome that was closed:

https://github.com/mailcow/mailcow-dockerized/issues/5932

I rolled back to version 2024-06a using git and the problem has disappeared. This definitely seems to be a bug. All of the syslog-ng errors are gone and I don't have to hack the dovecot.conf file to disable syslog-ng logging in order for mailcow to be usable. One problem is that the correct version isn't displayed after I log in, it still shows 2024-08a. Is there somewhere I can manually set the version back to 2024-06a so it matches the git rollback?

Anyhow, I would like to be running the latest version and it seems fairly certain that something was introduced in one of the releases after 2024-06a that are causing this bug.

Logs:

dovecot-mailcow-1 | 2024-07-19 17:37:33,702 WARN exited: syslog-ng (terminated by SIGILL (core dumped); not expected)
dovecot-mailcow-1 | 2024-07-19 17:37:34,924 WARN received SIGQUIT indicating exit request
dovecot-mailcow-1 | 2024-07-19 17:37:35,926 WARN stopped: syslog-ng (terminated by SIGILL (core dumped))
dovecot-mailcow-1 | 2024-07-19 17:37:37,932 WARN stopped: processes (terminated by SIGTERM)
dovecot-mailcow-1 | 2024-07-19 17:37:46,310 WARN exited: syslog-ng (terminated by SIGILL (core dumped); not expected)
dovecot-mailcow-1 | 2024-07-19 17:37:47,324 WARN received SIGQUIT indicating exit request
dovecot-mailcow-1 | 2024-07-19 17:37:48,406 WARN stopped: syslog-ng (terminated by SIGILL (core dumped))
dovecot-mailcow-1 | 2024-07-19 17:37:49,519 WARN stopped: processes (terminated by SIGTERM)
dovecot-mailcow-1 | 2024-07-19 17:37:55,808 WARN exited: syslog-ng (terminated by SIGILL (core dumped); not expected)
dovecot-mailcow-1 | 2024-07-19 18:37:57,264 WARN exited: syslog-ng (terminated by SIGILL (core dumped); not expected)
dovecot-mailcow-1 | 2024-07-19 18:37:58,534 WARN received SIGQUIT indicating exit request
dovecot-mailcow-1 | 2024-07-19 18:37:59,537 WARN stopped: syslog-ng (terminated by SIGILL (core dumped))
dovecot-mailcow-1 | 2024-07-19 18:38:01,542 WARN stopped: processes (terminated by SIGTERM)
dovecot-mailcow-1 | 2024-07-19 18:38:12,256 WARN exited: syslog-ng (terminated by SIGILL (core dumped); not expected)
dovecot-mailcow-1 | 2024-07-19 18:38:13,344 WARN exited: syslog-ng (terminated by SIGILL (core dumped); not expected)
dovecot-mailcow-1 | 2024-07-19 18:38:16,438 WARN exited: syslog-ng (terminated by SIGILL (core dumped); not expected)
dovecot-mailcow-1 | 2024-07-19 18:38:17,451 WARN received SIGQUIT indicating exit request
dovecot-mailcow-1 | 2024-07-19 18:38:17,520 WARN stopped: syslog-ng (terminated by SIGILL (core dumped))
dovecot-mailcow-1 | 2024-07-19 18:38:18,633 WARN stopped: processes (terminated by SIGTERM)

Steps to reproduce:

Update mailcow from 2024-6a to newer version.
Try to use mailcow like usual.
Observe constant syslog-ng crashes that make mailcow unusable.

Which branch are you using?

master

Which architecture are you using?

x86

Operating System:

Debian GNU/Linux 12 (bookworm)

Server/VM specifications:

6GB RAM, 4 CPU cores

Is Apparmor, SELinux or similar active?

yes

Virtualization technology:

Xen

Docker version:

27.3.1, build ce12230

docker-compose version or docker compose version:

v2.29.0

mailcow version:

2024-08a

Reverse proxy:

none

Logs of git diff:

Too long, not able to submit issue. I will post separately if requested.

"There was an error creating your issue: body is too long, body is too long (maximum is 65536 characters)."

Logs of iptables -L -vn:

Too long, not able to submit issue. I will post separately if requested.

"There was an error creating your issue: body is too long, body is too long (maximum is 65536 characters)."

Logs of ip6tables -L -vn:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 5083  371K MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */
   85 14924 f2b-sshd   6    --  *      *       ::/0                 ::/0                 multiport dports 22

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1539  369K MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */
 1539  369K DOCKER-USER  0    --  *      *       ::/0                 ::/0                
 1539  369K DOCKER-ISOLATION-STAGE-1  0    --  *      *       ::/0                 ::/0                
  756  272K ACCEPT     0    --  *      br-mailcow  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    2   160 DOCKER     0    --  *      br-mailcow  ::/0                 ::/0                
  781 96523 ACCEPT     0    --  br-mailcow !br-mailcow  ::/0                 ::/0                
    0     0 ACCEPT     0    --  br-mailcow br-mailcow  ::/0                 ::/0                
    0     0 ACCEPT     0    --  *      docker0  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 DOCKER     0    --  *      docker0  ::/0                 ::/0                
    0     0 ACCEPT     0    --  docker0 !docker0  ::/0                 ::/0                
    0     0 ACCEPT     0    --  docker0 docker0  ::/0                 ::/0                

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:110
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:143
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:993
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:995
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:4190
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::c  tcp dpt:80
    2   160 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::c  tcp dpt:443
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::f  tcp dpt:25
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::f  tcp dpt:465
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::f  tcp dpt:587

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  781 96523 DOCKER-ISOLATION-STAGE-2  0    --  br-mailcow !br-mailcow  ::/0                 ::/0                
    0     0 DOCKER-ISOLATION-STAGE-2  0    --  docker0 !docker0  ::/0                 ::/0                
86659   26M RETURN     0    --  *      *       ::/0                 ::/0                

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       0    --  *      br-mailcow  ::/0                 ::/0                
    0     0 DROP       0    --  *      docker0  ::/0                 ::/0                
42583   18M RETURN     0    --  *      *       ::/0                 ::/0                

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
86659   26M RETURN     0    --  *      *       ::/0                 ::/0                

Chain MAILCOW (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain f2b-sshd (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     0    --  *      *       2a01:799:293:ed00:fe4d:d4ff:fed3:57d3  ::/0                 reject-with icmp6-port-unreachable
   85 14924 RETURN     0    --  *      *       ::/0                 ::/0

Logs of iptables -L -vn -t nat:

Too long, not able to submit issue. I will post separately if requested.

"There was an error creating your issue: body is too long, body is too long (maximum is 65536 characters)."

Logs of ip6tables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  610 45364 DOCKER     0    --  *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     0    --  *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  759 71258 MASQUERADE  0    --  *      !br-mailcow  fd4d:6169:6c63:6f77::/64  ::/0                
    0     0 MASQUERADE  0    --  *      !docker0  fd00:dead:beef:c0::/80  ::/0                
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:110
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:143
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:993
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:995
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:4190
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:80
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:443
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:25
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:465
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:587

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   23  1840 RETURN     0    --  br-mailcow *       ::/0                 ::/0                
    0     0 RETURN     0    --  docker0 *       ::/0                 ::/0                
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:110 to:[fd4d:6169:6c63:6f77::b]:110
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:143 to:[fd4d:6169:6c63:6f77::b]:143
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:993 to:[fd4d:6169:6c63:6f77::b]:993
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:995 to:[fd4d:6169:6c63:6f77::b]:995
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:4190 to:[fd4d:6169:6c63:6f77::b]:4190
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:80 to:[fd4d:6169:6c63:6f77::c]:80
    2   160 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:443 to:[fd4d:6169:6c63:6f77::c]:443
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::f]:25
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::f]:465
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::f]:587

DNS check:

104.18.32.7
172.64.155.249
@DerLinkman
Copy link
Member

Any custom changes? Can not reproduce...

Do a git diff origin/master here pls

@DerLinkman
Copy link
Member

There is no issue on our side with our testing systems. There is something broken on your machine or due to some custom changes you made i guess

@DerLinkman
Copy link
Member

Hmm yes seems to be a issue with Xen Virtualized VMs, kvm VMs work fine.

Have to debug this but we might cut XEN Support if it is layered deeper down...

@DerLinkman DerLinkman mentioned this issue Oct 21, 2024
Open
5 tasks
@CouponCodeSwap

This comment was marked as spam.

Sign in to view
@DerLinkman
Copy link
Member

Dude please be a bit more structured.... don't paste any github prs here or what so ever.

Keep it clean and simple. No one can read that all

@CouponCodeSwap
Copy link
Author

Apologies. Hitting the Markdown <> button causes the browser to hang bad but the Comment button is still functional. Will try attaching git-diff as txt file.
git-diff.txt

@CouponCodeSwap
Copy link
Author

Another note that may be helpful pinpointing the problem: this server was incrementally upgraded from Debian 10 to 11 to 12 after Debian 12 was released. I wouldn't think that would be the cause of the issue since rolling back to 2024-06a works fine with the upgraded Debian but perhaps it is. @chriscroome

@DerLinkman
Copy link
Member

Apologies. Hitting the Markdown <> button causes the browser to hang bad but the Comment button is still functional. Will try attaching git-diff as txt file. git-diff.txt

Thanks,

could not see any difference except version differences as you rollback to 2024-06a.

@CouponCodeSwap
Copy link
Author

could not see any difference except version differences as you rollback to 2024-06a.

Thank you for looking into it. I was contemplating reinstalling OS, doing a fresh install of mailcow and then manually restoring from backups but that would be a major, time-consuming project because there are a lot of email accounts and settings to restore plus I also use the server for storage / backups and there are other things tied in that might break.

In your opinion, is it safe to assume that a fresh OS install and restore wouldn't fix the issue or is it worth a shot?

@DerLinkman
Copy link
Member

I don't know... i cannot reproduce this error anywhere even on a older machine so i'm clueless here...

@CouponCodeSwap
Copy link
Author

Is it possible to easily rollback only the dovecot container to the state it was in with the 2024-06c or 2024-06a (last known working version) release? If it is, I'd like to test doing that and if the issue persists, proceed one by one rolling back the other 3 containers that have sysleog-ng.

@DerLinkman
Copy link
Member

Yeah you can change the dovecot image version to the version of the working version by using docker-compose.override.yml

@CouponCodeSwap
Copy link
Author

Yeah you can change the dovecot image version to the version of the working version by using docker-compose.override.yml

Okay, I think I already tried that with dovecot. Will try again and continue with the other containers as well.

@CouponCodeSwap
Copy link
Author

mailcow/dovecot:1.28.2 = working
mailcow/dovecot:1.30 = broken

Surprisingly the issue is introduced with 1.30, I thought it would begin with the full version update to 2.0.

@DerLinkman
Copy link
Member

Versioning has nothing to say. We've retagged the images as we overwrote some older tags with that numbering.

@milkmaker
Copy link
Collaborator

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@milkmaker milkmaker added the stale Please update the issue with current status, unclear if it's still open/needed. label Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale Please update the issue with current status, unclear if it's still open/needed. unconfirmed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@CouponCodeSwap @DerLinkman @milkmaker