Restarting erlang server (RabbitMQ) continues to crash with memory alignment issues.

[ranjit6168]

Describe the bug
We are using RabbitMQ, an Erlang based application.
Erlang version : 26.2.5.2
OpenSSL version : 3.0.13 (used to build Erlang and consumed at runtime.)
RabbitMQ version : 3.13.7
Deployment type: containerized.

To Reproduce

• create podman image with the mentioned versions.
• Run the rabbitmq container and make connections from different podman services, which will try reconnection after rabbitmq is restarted.
• Redstart the podman rabbitmq container and it fails with core dump with error:

Startup logs:

2025-01-06 13:38:03.673247+00:00 [info] <0.254.0> Starting RabbitMQ 3.13.7 on Erlang 26.2.5.2 [emu]^[[0m
2025-01-06 13:38:03.673247+00:00 [info] <0.254.0> Copyright (c) 2007-2024 Broadcom Inc and/or its subsidiaries^[[0m
2025-01-06 13:38:03.673247+00:00 [info] <0.254.0> ^[[0m
2025-01-06 13:38:03.673247+00:00 [info] <0.254.0> Starting RabbitMQ 3.13.7 on Erlang 26.2.5.2 [emu]^[[0m
2025-01-06 13:38:03.673247+00:00 [info] <0.254.0> Copyright (c) 2007-2024 Broadcom Inc and/or its subsidiaries^[[0m
2025-01-06 13:38:03.673247+00:00 [info] <0.254.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com^[[0m

## RabbitMQ 3.13.7

########## Copyright (c) 2007-2024 Broadcom Inc and/or its subsidiaries

########## Licensed under the MPL 2.0. Website: https://rabbitmq.com

Erlang: 26.2.5.2 [emu]
TLS Library: OpenSSL - OpenSSL 3.0.13+SL_CCS-2.1.2+b 30 May 2024
Release series support status: see https://www.rabbitmq.com/release-information

Doc guides: https://www.rabbitmq.com/docs
Support: https://www.rabbitmq.com/docs/contact
Tutorials: https://www.rabbitmq.com/tutorials
Monitoring: https://www.rabbitmq.com/docs/monitoring
Upgrading: https://www.rabbitmq.com/docs/upgrade

Logs:

Config file(s): /etc/rabbitmq/advanced.config
/etc/rabbitmq/rabbitmq.conf

Starting broker...2025-01-06 13:38:03.701727+00:00 [info] <0.254.0> ^[[0m
2025-01-06 13:38:03.701727+00:00 [info] <0.254.0> node : rabbit@sample-rabbitmq^[[0m
2025-01-06 13:38:03.701727+00:00 [info] <0.254.0> home dir : /root^[[0m
2025-01-06 13:38:03.701727+00:00 [info] <0.254.0> config file(s) : /etc/rabbitmq/advanced.config^[[0m
2025-01-06 13:38:03.701727+00:00 [info] <0.254.0> : /etc/rabbitmq/rabbitmq.conf^[[0m
2025-01-06 13:38:03.701727+00:00 [info] <0.254.0> cookie hash : U++DQE0opnj5lfdsefsvNR5w==^[[0m
2025-01-06 13:38:03.701727+00:00 [info] <0.254.0> log(s) : ^[[0m
2025-01-06 13:38:03.701727+00:00 [info] <0.254.0> data dir : /rabbitvol/rabbitmq/data/rabbit@sample-rabbitmq^[[0m
2025-01-06 13:38:04.566218+00:00 [info] <0.254.0> Running boot step pre_boot defined by app rabbit^[[0m
2025-01-06 13:38:04.566335+00:00 [info] <0.254.0> Running boot step rabbit_global_counters defined by app rabbit^[[0m
2025-01-06 13:38:04.566571+00:00 [info] <0.254.0> Running boot step rabbit_osiris_metrics defined by app rabbit^[[0m
2025-01-06 13:38:04.566679+00:00 [info] <0.254.0> Running boot step rabbit_core_metrics defined by app rabbit^[[0m
2025-01-06 13:38:04.567040+00:00 [info] <0.254.0> Running boot step rabbit_alarm defined by app rabbit^[[0m
2025-01-06 13:38:04.688842+00:00 [info] <0.389.0> Memory high watermark set to 1024 MiB (1073741824 bytes) of 7685 MiB (8058503168 bytes) total^[[0m
2025-01-06 13:38:04.754571+00:00 [info] <0.391.0> Enabling free disk space monitoring (disk free space: 7044472832, total memory: 8058503168)^[[0m
2025-01-06 13:38:04.754665+00:00 [info] <0.391.0> Disk free limit set to 50MB^[[0m
2025-01-06 13:38:04.772842+00:00 [info] <0.254.0> Running boot step code_server_cache defined by app rabbit^[[0m
2025-01-06 13:38:04.773003+00:00 [info] <0.254.0> Running boot step file_handle_cache defined by app rabbit^[[0m
2025-01-06 13:38:04.829414+00:00 [info] <0.394.0> Limiting to approx 1048479 file handles (943629 sockets)^[[0m
2025-01-06 13:38:04.829640+00:00 [info] <0.395.0> FHC read buffering: OFF^[[0m
2025-01-06 13:38:04.829690+00:00 [info] <0.395.0> FHC write buffering: ON^[[0m
2025-01-06 13:38:04.830142+00:00 [info] <0.254.0> Running boot step worker_pool defined by app rabbit^[[0m
2025-01-06 13:38:04.830236+00:00 [info] <0.341.0> Will use 2 processes for default worker pool^[[0m
2025-01-06 13:38:04.830312+00:00 [info] <0.341.0> Starting worker pool 'worker_pool' with 2 processes in it^[[0m
2025-01-06 13:38:04.830630+00:00 [info] <0.254.0> Running boot step database defined by app rabbit^[[0m
2025-01-06 13:38:04.831629+00:00 [info] <0.254.0> Peer discovery: configured backend: rabbit_peer_discovery_classic_config^[[0m
2025-01-06 13:38:04.846457+00:00 [info] <0.254.0> Waiting for Mnesia tables for 30000 ms, 9 retries left^[[0m
2025-01-06 13:38:04.851258+00:00 [info] <0.254.0> Successfully synced tables from a peer^[[0m
2025-01-06 13:38:04.869338+00:00 [info] <0.254.0> Waiting for Mnesia tables for 30000 ms, 9 retries left^[[0m
2025-01-06 13:38:04.869574+00:00 [info] <0.254.0> Successfully synced tables from a peer^[[0m
2025-01-06 13:38:04.869747+00:00 [info] <0.254.0> Running boot step tracking_metadata_store defined by app rabbit^[[0m
2025-01-06 13:38:04.874189+00:00 [info] <0.407.0> Setting up a table for connection tracking on this node: tracked_connection^[[0m
2025-01-06 13:38:04.874294+00:00 [info] <0.407.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost^[[0m
2025-01-06 13:38:04.874381+00:00 [info] <0.407.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user^[[0m
2025-01-06 13:38:04.874456+00:00 [info] <0.407.0> Setting up a table for channel tracking on this node: tracked_channel^[[0m
2025-01-06 13:38:04.874519+00:00 [info] <0.407.0> Setting up a table for channel tracking on this node: tracked_channel_per_user^[[0m
2025-01-06 13:38:04.874602+00:00 [info] <0.254.0> Running boot step networking_metadata_store defined by app rabbit^[[0m
2025-01-06 13:38:04.874706+00:00 [info] <0.254.0> Running boot step feature_flags defined by app rabbit^[[0m
2025-01-06 13:38:04.880823+00:00 [info] <0.254.0> Running boot step codec_correctness_check defined by app rabbit^[[0m
2025-01-06 13:38:04.880953+00:00 [info] <0.254.0> Running boot step external_infrastructure defined by app rabbit^[[0m
2025-01-06 13:38:04.881082+00:00 [info] <0.254.0> Running boot step rabbit_event defined by app rabbit^[[0m
2025-01-06 13:38:04.888958+00:00 [info] <0.254.0> Running boot step rabbit_registry defined by app rabbit^[[0m
2025-01-06 13:38:04.894452+00:00 [info] <0.254.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit^[[0m
2025-01-06 13:38:04.895051+00:00 [info] <0.254.0> Running boot step rabbit_priority_queue defined by app rabbit^[[0m
2025-01-06 13:38:04.895191+00:00 [info] <0.254.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit^[[0m
2025-01-06 13:38:04.895237+00:00 [info] <0.254.0> Running boot step rabbit_queue_location_random defined by app rabbit^[[0m
2025-01-06 13:38:04.895348+00:00 [info] <0.254.0> Running boot step rabbit_auth_mechanism_ssl defined by app rabbitmq_auth_mechanism_ssl^[[0m
2025-01-06 13:38:04.895548+00:00 [info] <0.254.0> Running boot step kernel_ready defined by app rabbit^[[0m
2025-01-06 13:38:04.895599+00:00 [info] <0.254.0> Running boot step rabbit_sysmon_minder defined by app rabbit^[[0m
2025-01-06 13:38:04.896232+00:00 [info] <0.254.0> Running boot step rabbit_epmd_monitor defined by app rabbit^[[0m
2025-01-06 13:38:04.898983+00:00 [info] <0.415.0> epmd monitor knows us, inter-node communication (distribution) port: 25672^[[0m
2025-01-06 13:38:04.899116+00:00 [info] <0.254.0> Running boot step guid_generator defined by app rabbit^[[0m
2025-01-06 13:38:04.913404+00:00 [info] <0.254.0> Running boot step rabbit_node_monitor defined by app rabbit^[[0m
2025-01-06 13:38:04.913724+00:00 [info] <0.419.0> Starting rabbit_node_monitor (in ignore mode)^[[0m
2025-01-06 13:38:04.913832+00:00 [info] <0.254.0> Running boot step delegate_sup defined by app rabbit^[[0m
2025-01-06 13:38:04.926649+00:00 [info] <0.254.0> Running boot step rabbit_memory_monitor defined by app rabbit^[[0m
2025-01-06 13:38:04.926965+00:00 [info] <0.254.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit^[[0m
2025-01-06 13:38:04.927077+00:00 [info] <0.254.0> Running boot step core_initialized defined by app rabbit^[[0m
2025-01-06 13:38:04.933690+00:00 [info] <0.254.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit^[[0m
2025-01-06 13:38:04.934463+00:00 [info] <0.254.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit^[[0m
2025-01-06 13:38:04.934546+00:00 [info] <0.254.0> Running boot step rabbit_definitions_hashing defined by app rabbit^[[0m
2025-01-06 13:38:04.934645+00:00 [info] <0.254.0> Running boot step rabbit_exchange_parameters defined by app rabbit^[[0m
2025-01-06 13:38:05.020734+00:00 [info] <0.254.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit^[[0m
2025-01-06 13:38:05.028189+00:00 [info] <0.254.0> Running boot step rabbit_policies defined by app rabbit^[[0m
2025-01-06 13:38:05.028586+00:00 [info] <0.254.0> Running boot step rabbit_policy defined by app rabbit^[[0m
2025-01-06 13:38:05.035068+00:00 [info] <0.254.0> Running boot step rabbit_queue_location_validator defined by app rabbit^[[0m
2025-01-06 13:38:05.035222+00:00 [info] <0.254.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit^[[0m
2025-01-06 13:38:05.035322+00:00 [info] <0.254.0> Running boot step rabbit_quorum_queue defined by app rabbit^[[0m
2025-01-06 13:38:05.039456+00:00 [info] <0.254.0> Running boot step rabbit_stream_coordinator defined by app rabbit^[[0m
2025-01-06 13:38:05.040561+00:00 [info] <0.254.0> Running boot step rabbit_vhost_limit defined by app rabbit^[[0m
2025-01-06 13:38:05.040770+00:00 [info] <0.254.0> Running boot step recovery defined by app rabbit^[[0m
2025-01-06 13:38:05.232851+00:00 [info] <0.452.0> Starting message stores for vhost '/'^[[0m
2025-01-06 13:38:05.364670+00:00 [info] <0.452.0> Started message store of type transient for vhost '/'^[[0m
2025-01-06 13:38:05.413152+00:00 [info] <0.452.0> Started message store of type persistent for vhost '/'^[[0m
2025-01-06 13:38:05.755999+00:00 [info] <0.452.0> Recovering 6 queues of type rabbit_classic_queue took 620ms^[[0m
2025-01-06 13:38:05.756093+00:00 [info] <0.452.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms^[[0m
2025-01-06 13:38:05.758061+00:00 [info] <0.254.0> Will not seed default virtual host and user: have definitions to load...^[[0m
2025-01-06 13:38:05.767044+00:00 [info] <0.254.0> Running boot step notify_cluster defined by app rabbit^[[0m
2025-01-06 13:38:05.767514+00:00 [info] <0.254.0> Running boot step cluster_name defined by app rabbit^[[0m
2025-01-06 13:38:05.767864+00:00 [info] <0.538.0> Resetting node maintenance status^[[0m
2025-01-06 13:38:05.850544+00:00 [info] <0.538.0> Ready to start client connection listeners^[[0m
2025-01-06 13:38:06.724424+00:00 [info] <0.576.0> started TLS (SSL) listener on [::]:5671^[[0m
completed with 2 plugins.
2025-01-06 13:38:07.024862+00:00 [info] <0.538.0> Server startup complete; 2 plugins started.^[[0m
2025-01-06 13:38:07.024862+00:00 [info] <0.538.0> * rabbitmq_auth_backend_http^[[0m
2025-01-06 13:38:07.024862+00:00 [info] <0.538.0> * rabbitmq_auth_mechanism_ssl^[[0m
malloc(): unaligned tcache chunk detected
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed^M
[os_mon] memory supervisor port (memsup): Erlang has closed^M
/opt/rabbitmq/sbin/rabbitmq-server: line 139: 1053 Aborted (core dumped) start_rabbitmq_server "$@"
=INFO REPORT==== 6-Jan-2025::13:38:14.059188 ===
alarm_handler: {set,{{disk_almost_full,"/"},[]}}
=INFO REPORT==== 6-Jan-2025::13:38:14.113845 ===
alarm_handler: {set,{{disk_almost_full,"/rabbitvol"},[]}}

Expected behavior
Rabbitmq server should start gracefully.

Affected versions
The OTP versions that are affected by this bug - 26.2.5.2

[essen]

Please provide the erl_crash.dump file at a minimum if you got one.

[garazdawi]

malloc has detected that something is wrong, so some memory management in the VM or in a NIF has gone wrong. To solve this we are going to need a lot more information. What OS and HW architecture are you using? Please also provide the beam.smp executable used and the Linux core dump, or if not possible, open the core dump in gdb and do thread apply all bt full.