Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of remote exec in runnerv2 #724

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

adambabik
Copy link
Collaborator

@adambabik adambabik commented Jan 6, 2025

This PR refactors the runnerv2service package to speed up the performance of executing programs producing large output.

Test

cat a large file and pipe the output to /dev/null. As the execution goes through the server, the execution involves marshaling and unmarshaling to Protobuf.

This PR

time ./runme beta run --remote generate | cat - >/dev/null
./runme beta run --remote generate  0.70s user 0.64s system 33% cpu 3.973 total
cat - > /dev/null  0.01s user 0.06s system 1% cpu 3.972 total

Main

time ./runme beta run --remote generate | cat - >/dev/null
./runme beta run --remote generate  5.58s user 6.89s system 146% cpu 8.537 total
cat - > /dev/null  0.21s user 1.04s system 14% cpu 8.536 total

We have a speed increase of over 2x.

@adambabik adambabik force-pushed the adamb/speedup-runnerv2-execute branch from af7471b to 1ff4ba1 Compare January 12, 2025 07:05
@adambabik adambabik marked this pull request as ready for review January 12, 2025 09:44
@sourishkrout
Copy link
Member

@adambabik did you forget to check in a file? I can't seem to find the Runme task:

https://gist.github.com/sourishkrout/eabc0e47fd221866b118a2aad9f87190#file-01jhp3vbmryt06v9atw7m0zste-md

@adambabik
Copy link
Collaborator Author

@sourishkrout good point. I added it in 1daef79.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants