TiFlash panics with Too many open files in the cloud GCP env

[solotzg]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

• Topology: 3-tidb-16C32G / 9-tikv-16C64G-2000G / 3-pd-4C15G-50G / 2-tiflash-16C128G-500G
• Platform: GCP
• Unknown workloads

2. What did you expect to see? (Required)

• No panic

3. What did you see instead (Required)

Too many fd under tiflash process. The number of FD keeps growing, causing queries to fail and eventually tiflash to panic. Most of the FDs are related to sockets and a large amount of sockets are still open but can not be found in /proc/net.

sh-5.1# ls -l /proc/1/fd/ | wc -l
295268
sh-5.1# ls -l /proc/1/fd/ | grep "eventfd" | wc -l
98368
sh-5.1# ls -l /proc/1/fd/ | grep "eventpoll" | wc -l
98391
sh-5.1# ls -l /proc/1/fd/ | grep "socket" | wc -l
98492

Other

In the AWS environment, there is no such problem yet.

4. What is your TiFlash version? (Required)

v7.5.3

[solotzg]

After disabling mpp, the number of socket fd no longer continues to grow. There may be potential bugs in the implementation about mpp.

set global tidb_allow_fallback_to_tikv = "tiflash";
set global tidb_allow_mpp = 0;
set global tidb_allow_tiflash_cop = 1;

[solotzg]

grpc/grpc#32538 (comment)

For example, earlier versions of gRPC had an fd leak with the epollex poller, which was removed in v1.46. It may be too late to diagnose some of those years-old issues.

[solotzg]

grpc/grpc#20418 (comment)

The details were posted to the mailing list: Patch Releases for CVE-2023-4785, covering gRPC Core, C++, Python, and Ruby.

Lack of error handling in the TCP server in Google's gRPC starting version 1.23 on posix-compatible platforms (ex. Linux) allows an attacker to cause a denial of service by initiating a significant number of connections with the server. Note that gRPC C++ Python, and Ruby are affected, but gRPC Java, and Go are NOT affected.

The following set of releases contain the fix:

• 1.57.0 and later: https://github.com/grpc/grpc/releases/tag/v1.57.0
• 1.56.2: https://github.com/grpc/grpc/releases/tag/v1.56.2
• 1.55.3: https://github.com/grpc/grpc/releases/tag/v1.55.3
• 1.54.3: https://github.com/grpc/grpc/releases/tag/v1.54.3
• 1.53.2: https://github.com/grpc/grpc/releases/tag/v1.53.2