Too many open files after a while

[Nitaaq]

Hello,
I've been running a nextJS app under passenger installed using WHM EasyApache 4,

The main process PassengerAgent is incrementally opening file descriptors without closing them, leads to reaching the ulimit after a while

Looked up the files they were named buffer.(number) with 0 bytes in them

Tried looking up why passenger is creating these files at the first place, couldn't find anything in the documentation referring to these files

Can you please help with whatever is going on

OS: AlmaLinux v8.10.0 STANDARD
Passenger Version: 6.0.23

[Nitaaq]

after more digging found out that these files are created like this:

[ D3 2024-10-07 11:11:05.8215 604538/T2j Ser/HttpChunkedBodyParser.h:162 ]: [Client 14] ChunkedBodyParser: parsing 4048 of 14786 bytes of remaining chunk data; 10738 now remaining
[ D3 2024-10-07 11:11:05.8215 604538/T2j Ser/FileBufferedChannel.h:1416 ]: [FBC 0x14fe3c000d20] Feeding 4048 bytes
[ D3 2024-10-07 11:11:05.8215 604538/T2j Ser/FileBufferedChannel.h:486 ]: [FBC 0x14fe3c000d20] pushBuffer() completed: nbuffers = 41, bytesBuffered = 133520
[ D3 2024-10-07 11:11:05.8215 604538/T2j Ser/FileBufferedChannel.h:838 ]: [FBC 0x14fe3c000d20] Switching to in-file mode
[ D3 2024-10-07 11:11:05.8215 604538/T2j Ser/FileBufferedChannel.h:888 ]: [FBC 0x14fe3c000d20] Writer: creating file /var/run/ea-ruby27-passenger/buffer.2090110035
[ D2 2024-10-07 11:11:05.8215 604538/T2j age/Cor/Con/ForwardResponse.cpp:1018 ]: [Client 42-14] Application is sending response data quicker than the on-disk buffer can keep up with (currently buffered 133520 bytes). Throttling application socket

the file is being created and opened, but never closed again.

[KibetMasiror]

I'm also getting the same error when I host my Nuxt SSR application.

An operating system error occurred while preparing to spawn an application process: Too many open files (errno=24)

Could not spawn process for application /server: An operating system error occurred while preparing to spawn an application process: Cannot open /var/run/ea-ruby27-passenger/passenger.spawn.XXXX4WWQGH/envdump: Too many open files (errno=24)

[KibetMasiror]

@Nitaaq Were you able to find a solution?

[Nitaaq]

@KibetMasiror been doing some digging, turns out whenever the ssr output is too big passenger defers to on-disk buffer,

This buffer opens a file descriptor but for some reason doesnt close it,

possible fixes is to make your SSR output less in size, or do a cron job that restarts apache/nginx which is almost instant depending on your configuration

[KibetMasiror]

@Nitaaq Thanks for your response. Will definitely try it out.

Could the problem also be caused by high traffic such that each visit spawns a new process then fails to close it for some reason (memory leaks caused by conflicting processes)?

[Nitaaq]

@KibetMasiror we do have high traffic but the process is not getting killed, no memory leaks whatsoever, my issue is more likely is a file descriptor leak than a memory leak.

you can check your /var/run/ea-ruby27-passenger for buffer.[random number] files, and check your open files from the main PassengerAgent process by determining your pid with this command:
ps ax | grep "Passenger core"

and running this command to see how much open files you have at the moment
ls /proc/[PassengerAgent pid]/fd -l | wc -l

if this number keeps increasing over time it will eventually reach the limit specified in here
cat /proc/[PassengerAgent pid]/limits | grep "Max open files"

[KibetMasiror]

@Nitaaq Thank you so much for your assistance