Add grpc window size config and change the default value #1239
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add
GrpcInitialWindowSize
andGrpcInitialConnWindowSize
and change their default value from1GiB
to128MiB
.The default value
1GiB
may cause a TiDB OOM issue, see pingcap/tidb#49353.The initial default value came from this PR pingcap/tidb#3377 where the author referred to pingcap/tidb#3377 (comment) as the evidence to justify the
1GiB
default value. In this referenced document, thehttp2-window
size is set to1MiB
andhttp2-connection-window
is set to1GiB
. However, this document did not clearly describe how the test was done. I guess the key factor in getting faster results is changinghttp2-window
from64KiB
to1MiB
, not changinghttp2-connection-window
to1GiB
because at the top of the doc said the speed was just4MiB/s
if not using http2.There is a lot of evidence that
1GiB
is too large.grpc-go
will use the dynamic window size and the maximum of this dynamic window size is16MiB
. https://github.com/grpc/grpc-go/blob/dadbbfa2863a67ed640aac924b7b7fd18b50a429/internal/transport/bdp_estimator.go#L30GrpcInitialWindowSize
andGrpcInitialConnWindowSize
are2MiB
and32MiB
, respectively. The maximum of these two values is64MiB
. https://github.com/cockroachdb/cockroach/blob/master/pkg/rpc/settings.go#L69-L70. (rpc: permit gRPC stream window sizes up to 64 MB cockroachdb/cockroach#111255)As a result, changing both default values to
128MiB
will not likely cause performance issues but will likely reduce many TiDB OOM issues.