Pool: java.lang.OutOfMemoryError

[vingar]

Hello,

We have recently observed that some pools are restarting frequently due to a java.lang.OutOfMemoryError. The log file seems to suggest a memory leak issue:

23 Aug 2024 20:30:29 (dc269_2) [] Restarting due to fatal JVM error: java.lang.OutOfMemoryError: Direct buffer memory
23 Aug 2024 20:30:29 (dc269_2) [] Pool mode changed to disabled (fetch, store, stage, p2p-client, p2p-server, dead): Shutdown
23 Aug 2024 20:30:29 (dc269_2) [door
.1@dcdb02Domain
nfsv4.1 PoolAcceptFile 00005C8C04CCDFBD4BF2A16EFC38281B1D05] Failed to update namespace: Request to [>PnfsManager@local] timed out.. Retrying in 15 s
23 Aug 2024 20:30:30 (dc269_2) [] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.

The log mentions NFSv4.1 operations, which I was unable to confirm during the period with the billing records: the protocols used were Http-1.1, RemoteHttpsDataTransfer-1.1 and Xrootd-5.0. I am attaching the pool layout configuration and the log file.

Thank you for assisting us in resolving this issue

Settings:
dCache: 9.2.17
Java: openjdk 11.0.22 2024-01-16 LTS
OS: Red Hat Enterprise Linux release 8.9 (Ootpa)

log+layout.zip