fix: patch artifact download takes much longer than it should on Windows

[bryanoltman]

Description

Users have reported extremely long (thousands of seconds) download times of release artifacts when creating a patch on Windows.

Example: https://discord.com/channels/1030243211995791380/1293125397394751581

This seems to be mostly affecting Windows users but I'm not confident it is a Windows-only issue.

[bryanoltman]

We've released an update to the shorebird tool with better diagnostics (v1.4.1). If anyone experiencing this can run shorebird doctor -v with this new version and share the output, that would be very helpful.

[bryanoltman]

I'm currently able to reproduce this on my windows machine. I:

1. Downgraded to Flutter 3.19.6 as my non-Shorebird flutter
2. Created a new app
3. Ran shorebird release android (no issues)
4. Ran shorebird patch android and am seeing very very slow (but progressing) downloads.

I'm going to try re-upgrading my flutter to see if that changes anything.

Sample output:

PS C:\Users\bryan\Documents\sandbox\windows_old_flutter_test
$ shorebird patch android
✓ Fetching apps (0.2s)
✓ Fetching releases (0.1s)
Which release would you like to patch?
Which release would you like to patch? 1.0.0+1
✓ Fetching aab artifact (0.1s)
✓ Downloading aab (100%) (1.8s)
✓ Building patch with Flutter 3.24.3 (5a6dc546e8) (8.7s)
✓ Verifying patch can be applied to release (35ms)
✓ Fetching release artifacts (4.7s)
⠦ Downloading release artifact 1/3 (28%)... (135.4s)

I killed the process shortly after this. The progress percentage had been slowly creeping up.

[bryanoltman]

The issue above was using powershell. Can confirm that git bash does not have this issue (patching works as expected).

[bryanoltman]

Frustratingly, after trying this in git bash and seeing that it worked fine there, I'm not able to reproduce it in Powershell

[bryanoltman]

Correction: if I don't re-upgrade, the issue is reproducible in both powershell and git bash

[bryanoltman]

AAB url, from which I have no trouble downloading a file:

[HTTP] GET https://storage.googleapis.com/shorebird_release_artifacts/a9d8c7a0-1e89-4ebf-b6d7-fdf8cb34f3a8%2Fandroid%2Faab%2F115801%2Fapp-release.aab?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=patch-artifacts-storage-bucket%40code-push-prod.iam.gserviceaccount.com%2F20241022%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20241022T212330Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=94c64f12865ebc051e270b204dead52da37b3b6e2d4de0fb598fe538023f39ea19473f5a6e05ec871f1480308c3ed369e98329fd3b3eb9f67e42bf675734af4291d9c6efeb4a96a6c78c840c422f7c2b3960f7ac16ddd224aaa12e77195c710ddd87de680ddf2a2b02f7a74b4359d4f8b02bffea0c422e626a0812cc10133719715cbcd7e958a846c5d069cbb2ddafccb84a8a35827dc7842f98dbbf52bda5d3d26bd96107b85ff9de2006c1889a7c50c13ddf2bcc20c1a8373aa19892a2ad4c56900097a2725f145dc89a3f122fd4f522a8f267f74d3e9bc4ee6b07f56ca10951e79618582882b914345a698713e08864277eaf62e647460f35681f56b90d2c
✓ Downloading aab (100%) (2.8s)

libapp.so artifact url, which downloads super slowly:

[HTTP] GET https://storage.googleapis.com/shorebird_release_artifacts/a9d8c7a0-1e89-4ebf-b6d7-fdf8cb34f3a8%2Fandroid%2Farm%2F115801%2Flibapp.so?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=patch-artifacts-storage-bucket%40code-push-prod.iam.gserviceaccount.com%2F20241022%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20241022T212342Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=034ca163f466fdaba7d4d8999f4295774d531b0fd3429e4d56b1d4b2aa72104001f92ade72b7576f0543af5f7dedd7c92ce9c2de78b70580cba06d4ef7a1a30e1f43a93cfc8a0ffe641f5ce44b50eb109b8f58a2c7bd581d89340faf789f3ece9eec9fedaf264601dca4372695c69d97364f438b2076005fedb72ed6bf6f212917b66a6c69e6c29a9bfd52567354566987e1d30d2a0b115d44f6c4b26e67e8d9c0f5d147059a37abbf4e928532bd7570d2fa07f58678c7f08b4345b1294315315fa7c38110a2ed802dba68bcf40dcdeb080d9716843a058a668d5049553e1311d5aeaae7239ccda9c665bede3aa2fd53e52f06b594c1a48c3f6ad42cc7ebfb8b

Despite the aab being an order of magnitude larger, it downloaded in <3s, while the libapp.so has downloaded 37% in 3 minutes.

[eseidel]

Do the URLs "warm up" in any way? e.g. if you try repeatedly to download from the libapp.so url does it ever get faster?

Could it relate to the extension of the file being downloaded (could windows defender being doing something because it's a .so file?)

[bryanoltman]

I've updated the patch command code to attempt to download the aab at different points in the patch process. After a few runs, the throughline seems to be that it goes much slower after the build. This continues to be the case when I remove our invocation of the system Flutter's flutter pub get command (which I tried because this seems like it might be related to the system flutter version).

[bryanoltman]

Looking in Task Manager, there does not appear to be any obvious resource consumption. I'm using 10% of CPU, 57% of memory, and there's basically no network traffic. There does seem to be a dart.exe process running in efficiency mode though. Looking into that now.

[bryanoltman]

Update: In task manager, if I change the priority of the dart.exe task from "normal" to "high", the download suddenly picks up speed

[eseidel]

a Dart VM trace (from observatory or with command line flags) might still help you here, as you might be able to see where the hangs are happening. Is it that the networking stack isn't getting back to us in a timely manner? Is it that we're not getting scheduled to process the results? Is it that the packet sizes are being forced to being smaller than expected, etc.

[bryanoltman]

A couple more updates:

1. I found that disabling Windows Defender's real time protection entirely fixed this issue
   
2. After installing "2024-10 Cumulative Update Preview for Windows 11 Version 23H2 for x64-based Systems (KB5044380)", I am no longer able to reproduce the issue, even with defender's real time protection enabled.

[bryanoltman]

KB5045991 seems like it might be the corresponding update on Windows 10, will update when I can confirm

[bryanoltman]

Another report of disabling Windows Defender's real-time protection solving the problem https://discord.com/channels/1030243211995791380/1299963798060404829/1300498923269394483