Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Buffer Size Configuration and Warning Logs for Sync Optimization #5497

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

YinhaoHu
Copy link

@YinhaoHu YinhaoHu commented Jan 2, 2025

close #5492

@CLAassistant
Copy link

CLAassistant commented Jan 2, 2025

CLA assistant check
All committers have signed the CLA.

@YinhaoHu YinhaoHu closed this Jan 2, 2025
@YinhaoHu YinhaoHu deleted the fix/assignable-buffer-size-in-sync branch January 2, 2025 16:46
@YinhaoHu YinhaoHu restored the fix/assignable-buffer-size-in-sync branch January 2, 2025 16:50
@YinhaoHu YinhaoHu reopened this Jan 2, 2025
@YinhaoHu YinhaoHu force-pushed the fix/assignable-buffer-size-in-sync branch from 9872447 to 913e5a9 Compare January 2, 2025 16:54
@zhijian-pro
Copy link
Contributor

@YinhaoHu Thank you very much for your contribution, but we think the following approach would be more appropriate.
The issue lies in our sync download logic where we shouldn't be using chunk.NewOffPage to allocate memory.

p := chunk.NewOffPage(int(size))

In this line of code
downer := newParallelDownloader(src, key, size, 10<<20, concurrent)

we have hard-coded a block size of 10M, so we can directly use a fixed-size sync.pool to allocate memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sync Process Stalls with High Thread Count Due to Buffer Limitations
3 participants