Nethermind archive sync always loses speed in few hours of uptime

[mksh]

Description
Nethermind archive sync becomes very slow after few hours.

Steps to Reproduce
Start Nethermind 1.28+ node on dedicated server, wait until it syncs to 14M+ block, expect it will be syncing well further. Config is the following

      - --config
      - mainnet_archive
      - --datadir
      - "/.nethermind"
      - --Metrics.Enabled
      - "true"
      - --Metrics.ExposePort
      - "6060"
      - --JsonRpc.Enabled
      - "true"
      - --Init.WebSocketsEnabled
      - "true"
      - --JsonRpc.WebSocketsPort
      - "8546"
      - --JsonRpc.Host
      - "0.0.0.0"
      - --JsonRpc.Port
      - "8545"
      - --Network.P2PPort
      - "13030"
      - --Network.DiscoveryPort
      - "13030"
      - --JsonRpc.EngineHost
      - "0.0.0.0"
      - --JsonRpc.EnginePort
      - "8551"
      - --JsonRpc.JwtSecretFile
      - "/.jwt/secret"
      - --KeyStore.KeyStoreDirectory
      - "/.nethermind/keystore"
      - --Init.LogDirectory
      - "/.nethermind/logs"
      - --log
      - "INFO"

Actual behavior
The server loses sync speed and needs about 10-60 seconds to import each one (1) new block

Expected behavior
The syncing speed should keep up to be at least 5-10 blocks per second as it happens on the start

Screenshots

Desktop (please complete the following information):
Please provide the following information regarding your setup:

• Operating System: Linux
• Version: Ubuntu 24.04
• Installation Method: Docker
• Consensus Client: Lighthouse 5.3.0

Additional context
Server health seems to be fine, all the disks smarctl output shows no error

Logs
The logs after restart

18 Oct 09:41:05 | Processed      14807194... 14807204  |  1,047.43 ms  |  slot      1,048 ms |⛽ Gas gwei: 18.27 .. 19.04 (22.59) .. 150.00 
18 Oct 09:41:05 | - Blocks 11           159.60 MGas    |  2,074    txs |  calls  5,963 (379) | sload  28,676 | sstore  7,294 | create  20 
18 Oct 09:41:05 | - Block throughput    152.37 MGas/s  |   1980.08 t/s |         10.50 Blk/s | recover 5,028 | process 2,000 
18 Oct 09:41:06 | Processed      14807205... 14807219  |  1,046.81 ms  |  slot      1,048 ms |⛽ Gas gwei: 23.78 .. 23.78 (23.78) .. 23.78 
18 Oct 09:41:06 | - Blocks 15           252.02 MGas    |  2,523    txs |  calls 10,851 (632) | sload  50,377 | sstore 12,843 | create  47 
18 Oct 09:41:06 | - Block throughput    240.75 MGas/s  |   2410.18 t/s |         14.33 Blk/s | recover 5,013 | process 2,000 
18 Oct 09:41:07 | Processed      14807220... 14807229  |  1,106.97 ms  |  slot      1,108 ms |⛽ Gas gwei: 21.73 .. 21.73 (21.73) .. 21.73 
18 Oct 09:41:07 | - Blocks 10           144.13 MGas    |  1,733    txs |  calls  5,488 (416) | sload  24,692 | sstore  6,601 | create  67 
18 Oct 09:41:07 | - Block throughput    130.21 MGas/s  |   1565.54 t/s |          9.03 Blk/s | recover 5,003 | process 2,000 

The logs in few hours when issue hits:

18 Oct 09:12:46 | Processed            14794206        | 32,434.46 ms  |  slot     32,435 ms |⛽ Gas gwei: 33.96 .. 33.96 (35.72) .. 66.15 
18 Oct 09:12:46 | - Block                16.02 MGas    |    208    txs |  calls    596 ( 63) | sload   2,217 | sstore    788 | create   1 
18 Oct 09:12:46 | - Block throughput      0.49 MGas/s  |      6.41 t/s |          0.03 Blk/s | recover 5,049 | process 2,000 
18 Oct 09:13:14 | Processed            14794207        | 28,038.21 ms  |  slot     28,038 ms |⛽ Gas gwei: 34.28 .. 34.28 (34.75) .. 35.24 
18 Oct 09:13:14 | - Block                 0.57 MGas    |      7    txs |  calls     23 (  3) | sload      90 | sstore     26 | create   0 
18 Oct 09:13:14 | - Block throughput      0.02 MGas/s  |      0.25 t/s |          0.04 Blk/s | recover 5,048 | process 2,000 
18 Oct 09:13:56 | Processed            14794208        | 37,093.51 ms  |  slot     37,094 ms |⛽ Gas gwei: 31.25 .. 31.25 (41.87) .. 92.23 
18 Oct 09:13:56 | - Block                 3.33 MGas    |     48    txs |  calls    169 (  8) | sload     617 | sstore    193 | create   0 
18 Oct 09:13:56 | - Block throughput      0.09 MGas/s  |      1.29 t/s |          0.03 Blk/s | recover 5,047 | process 2,000 
18 Oct 09:14:22 | Processed            14794209        | 28,359.79 ms  |  slot     28,360 ms |⛽ Gas gwei: 27.00 .. 27.21 (29.92) .. 51.20 
18 Oct 09:14:22 | - Block                30.00 MGas    |    294    txs |  calls  1,347 ( 97) | sload   8,098 | sstore  1,210 | create   3 
18 Oct 09:14:22 | - Block throughput      1.06 MGas/s  |     10.37 t/s |          0.04 Blk/s | recover 5,046 | process 2,000 
18 Oct 09:14:53 | Processed            14794210        | 28,265.53 ms  |  slot     28,266 ms |⛽ Gas gwei: 29.70 .. 30.49 (40.90) .. 1,170.40 
18 Oct 09:14:53 | - Block                15.82 MGas    |    198    txs |  calls    639 ( 58) | sload   2,535 | sstore    855 | create   2 
18 Oct 09:14:53 | - Block throughput      0.56 MGas/s  |      7.00 t/s |          0.04 Blk/s | recover 5,045 | process 2,000 
18 Oct 09:15:36 | Processed            14794211        | 47,401.73 ms  |  slot     47,402 ms |⛽ Gas gwei: 30.91 .. 30.91 (35.49) .. 430.00 
18 Oct 09:15:36 | - Block                19.07 MGas    |    235    txs |  calls    686 ( 38) | sload   2,810 | sstore    930 | create   4 
18 Oct 09:15:36 | - Block throughput      0.40 MGas/s  |      4.96 t/s |          0.02 Blk/s | recover 5,044 | process 2,000 
18 Oct 09:16:09 | Processed            14794212        | 17,816.42 ms  |  slot     17,817 ms |⛽ Gas gwei: 31.92 .. 31.92 (34.07) .. 55.77 
18 Oct 09:16:09 | - Block                 2.61 MGas    |     17    txs |  calls     88 ( 12) | sload     297 | sstore    125 | create   3 
18 Oct 09:16:09 | - Block throughput      0.15 MGas/s  |      0.95 t/s |          0.06 Blk/s | recover 5,043 | process 2,000 
18 Oct 09:16:27 | Processed            14794213        | 35,614.34 ms  |  slot     35,614 ms |⛽ Gas gwei: 27.83 .. 27.90 (30.53) .. 56.00 
18 Oct 09:16:27 | - Block                 9.27 MGas    |    125    txs |  calls    318 ( 25) | sload   1,411 | sstore    444 | create   1 
18 Oct 09:16:27 | - Block throughput      0.26 MGas/s  |      3.51 t/s |          0.03 Blk/s | recover 5,042 | process 2,000 
18 Oct 09:17:00 | Processed            14794214        | 30,863.37 ms  |  slot     30,864 ms |⛽ Gas gwei: 27.41 .. 27.41 (32.76) .. 130.00 
18 Oct 09:17:00 | - Block                 5.85 MGas    |     61    txs |  calls    178 ( 12) | sload   1,039 | sstore    454 | create   0 
18 Oct 09:17:00 | - Block throughput      0.19 MGas/s  |      1.98 t/s |          0.03 Blk/s | recover 5,041 | process 2,000 
18 Oct 09:17:29 | Processed            14794215        | 26,971.07 ms  |  slot     26,971 ms |⛽ Gas gwei: 25.39 .. 25.39 (26.73) .. 37.35 
18 Oct 09:17:29 | - Block                 6.88 MGas    |     73    txs |  calls    405 ( 22) | sload   1,425 | sstore    355 | create   0 
18 Oct 09:17:29 | - Block throughput      0.25 MGas/s  |      2.71 t/s |          0.04 Blk/s | recover 5,040 | process 2,000 
18 Oct 09:17:54 | Processed            14794216        | 27,383.83 ms  |  slot     27,384 ms |⛽ Gas gwei: 24.24 .. 24.24 (35.83) .. 929.84 
18 Oct 09:17:54 | - Block                29.99 MGas    |    308    txs |  calls  1,094 ( 52) | sload   4,308 | sstore  1,385 | create   1 
18 Oct 09:17:54 | - Block throughput      1.10 MGas/s  |     11.25 t/s |          0.04 Blk/s | recover 5,039 | process 2,000 
18 Oct 09:18:32 | Processed            14794217        | 38,355.80 ms  |  slot     38,356 ms |⛽ Gas gwei: 26.58 .. 26.58 (27.43) .. 80.00 

As it is seen from the logs, importing new blocks takes more than 12 seconds so in theory this server will never sync. However, we were able to make it moving with manual restarts 2-3 times per day. It would be great to figure out the cause of this behavior, let me know if you need any additional information.