Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A large volume of requests combined with slow response time will significantly increase the memory usage of the PD #8882

Open
okJiang opened this issue Dec 9, 2024 · 3 comments
Open

A large volume of requests combined with slow response time will significantly increase the memory usage of the PD #8882

okJiang opened this issue Dec 9, 2024 · 3 comments
Labels
report/customer Customers have encountered this bug. type/enhancement The issue or PR belongs to an enhancement.

Comments

@okJiang
Copy link
Member

okJiang commented Dec 9, 2024 •
edited
Loading

Bug Report

What did you do?

In a 1 million region cluster, simultaneously start 600,000 ScanRegions requests to PD.

What did you expect to see?

Memory is maintained within an acceptable range.

What did you see instead?

Memory usage has increased by several times.

Image

What version of PD are you using (pd-server -V)?
@okJiang okJiang added the type/bug The issue is confirmed as a bug. label Dec 9, 2024
@okJiang
Copy link
Member Author

okJiang commented Dec 9, 2024

The reason for the increase in memory is a bug in gRPC. grpc/grpc-go#7261

It is unreasonable to make such large requests to such a massive cluster. This issue is temporarily noted for record-keeping. Perhaps significant improvements can be achieved after implementing #8678.

@okJiang okJiang changed the title A large volume of HTTP requests combined with slow response time will significantly increase the memory usage of the PD A large volume of requests combined with slow response time will significantly increase the memory usage of the PD Dec 12, 2024
@okJiang okJiang mentioned this issue Dec 16, 2024
Closed
@seiya-annie
Copy link

/report customer

@ti-chi-bot ti-chi-bot bot added the report/customer Customers have encountered this bug. label Dec 18, 2024
@okJiang okJiang added type/enhancement The issue or PR belongs to an enhancement. and removed type/bug The issue is confirmed as a bug. severity/major may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 affects-8.1 This bug affects the 8.1.x(LTS) versions. may-affects-8.5 labels Dec 30, 2024
@okJiang
Copy link
Member Author

okJiang commented Jan 3, 2025

Maybe we should set maxConcurrentStreams in grpc to limit the goroutines. Ref https://github.com/grpc/grpc-go/blob/66ba4b264d26808cb7af3c86eee66e843472915e/server.go#L1023-L1042.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
report/customer Customers have encountered this bug. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jebter @okJiang @seiya-annie