Fluent Bit 3.1.9 keeps old FIFO pipe files open and creates new ones after each hot reload

[mpuum]

Bug Report

Describe the bug

When using tail input plugin and hot reload with SIGHUP, Fluent Bit clears monitored files based on Ignore_Older config but keeps old pipe FIFO files open and creates new ones after each hot reload. This causes too many files open issue after while, depending reload frequency.

To Reproduce

• Key configuration parameters:

  [SERVICE]
  Hot_Reload On

  [INPUT]
  name tail
  Ignore_Older 1h
  path /var/log/logfile*.log
  read_from_head true
  DB /opt/fluent-bit/DB.db

• Steps to reproduce the problem:

  • Create new log files
  • Use SIGHUP to reload config several times
  • Observe with lsof increased number of open FIFO files after each reload

# ulimit -n
1024

Nov  1 00:00:01 server-1 fluent-bit[9694]: pipe: Too many open files
Nov  1 00:00:01 server-1 fluent-bit[9694]: [2024/11/01 00:00:01] [error] [reload] creating flb context is failed. Reloading is halted

# lsof -p 9694 | wc -l
1068

# lsof -p 9694 | grep pipe | wc -l
998

# lsof -p 9694 | grep inode | wc -l
18

Your Environment

• Version used: fluent-bit-3.1.9-1.x86_64.rpm
• Operating System and version: RHEL 8.10

Additional context
Noticed with workaround used for the issue #9021 "Broken "Ignore Older" setting in in_tail plugin"