🚨RESOLVED: Network disruption🚨

[redfish31]

If you are looking for help, please confirm the following...

• I confirm I have searched the Docs, Codemagic Sample Projects, and GitHub Discussions.
• I confirm the software versions on my local computer match the ones in CI environment (e.g. Xcode version, Gradle version). See here
• I confirm I am able to do a fresh clone of the repository and build the project on my local computer.

Which mobile framework are you using?

Flutter (Dart)

Steps to reproduce

Try to installing SDK

Expected results

Download & Install

Actual results

== Use Flutter channel stable ==
> curl --fail --show-error --silent --output flutter_macos_arm64_3.19.3-stable.zip --connect-timeout 10.00 --max-time 45.00 https://storage.googleapis.com/flutter_infra_release/releases/stable/macos/flutter_macos_arm64_3.19.3-stable.zip
curl: (28) Operation timed out after 45003 milliseconds with 276924723 out of 1167817719 bytes received

Downloading Flutter release timed out, try again...
> curl --fail --show-error --silent --output flutter_macos_arm64_3.19.3-stable.zip --connect-timeout 10.00 --max-time 45.00 https://storage.googleapis.com/flutter_infra_release/releases/stable/macos/flutter_macos_arm64_3.19.3-stable.zip
curl: (28) Operation timed out after 45003 milliseconds with 545429997 out of 1167817719 bytes received

Downloading Flutter release timed out, try again...
> curl --fail --show-error --silent --output flutter_macos_arm64_3.19.3-stable.zip --connect-timeout 10.00 --max-time 45.00 https://storage.googleapis.com/flutter_infra_release/releases/stable/macos/flutter_macos_arm64_3.19.3-stable.zip
curl: (28) Operation timed out after 45001 milliseconds with 118802687 out of 1167817719 bytes received

Downloading Flutter release timed out

Build failed :|
Downloading Flutter release timed out

Build id (optional)

No response

[mikhail-tokarev]

We are seeing an increased number of network errors when interacting with Google Cloud Storage and our team is looking how we can improve things.

[helinanever]

Unfortunately, there have been new incidents today, causing both timeouts and Something Went Wrong errors. Please see the status page here: https://status.codemagic.io/

[ggirotto]

@helinanever the issue is marked as resolved in the status page but the issues are still happening

[Imgkl]

Yes. We are facing this issue.

[peer-f]

Facing this issue too. It worked fine in the morning but now it fails every time.

[mrruby]

It's still the issue for me

[markbreuss]

We are still seeing timed out builds.

Build ID:
66019467d09bc98f417de6f6

Screenshot:

[ChrisElliotUK]

Same for us.

Build ID:
660195155f9460b428104153

[mikhail-tokarev]

Unfortunately the network performance issue has returned. We are working with our data center partner to resolve the issue. Please see the status page here: https://status.codemagic.io/

Additionally we are trying to provide more machines in other data centers to reduce the impact of the issue.

Thank you for your patience!

[mikhail-tokarev]

we are still waiting for the update from our data center partners :(((

[ChrisElliotUK]

Annoyingly this issue seems to be back again this morning. We managed two builds before getting a timeout.
build id: 6602945540a27391b82e2b84

[warpaltarpers]

I think if we keep telling Codemagic that it's not working, the off-site data center will get fixed faster 🤔

[mikhail-tokarev]

thank you for your understanding!
However right now we are receiving confusing information and I need your help 🙏 Can somebody please share build id failed with network issue after 18:30UTC?

[lstimacay]

@mikhail-tokarev Here it is: 66031729b11f45d5560ed4e3

Failed with:

== Use Flutter version 3.3.9 ==
> curl --fail --show-error --silent --output flutter_macos_arm64_3.3.9-stable.zip --connect-timeout 10.00 --max-time 120.00 https://storage.googleapis.com/flutter_infra_release/releases/stable/macos/flutter_macos_arm64_3.3.9-stable.zip
curl: (28) Operation timed out after 120006 milliseconds with 514787713 out of 1007490932 bytes received

Downloading Flutter release timed out

Build failed :|
Downloading Flutter release timed out

[MCD-Daniel]

Hi @mikhail-tokarev This is my last successful build Id 6602ca18597e0501c035d172
But now the same error occurs again and I cannot build.
This is my latest failed build Id 66031798a88d5a056266215e

[warpaltarpers]

If it's at all helpful for triage, we have had 3 builds succeed:

• Started at 19:01UTC: 66031b8263c492b39d7b552d
• Started at 19:17UTC: 66031d1328e00f2f74a2e10e
• Started at 19:28UTC: 660321c3538a95a262e29dd5

[burekas7]

So what should we need to do here? Just waiting and to try later again?

[tony123S]

just tried, still not working...

[lucas-sesti]

Just tried now, still not working too..

[ggirotto]

Please stop spamming the discussion. Commenting "me too", "not working" or anything related WILL NOT fix the issue. Codemagic team is aware of the issue and they'll post any update once they have it.

[burekas7]

Unfortunately the network performance issue has returned. We are working with our data center partner to resolve the issue. Please see the status page here: https://status.codemagic.io/

Additionally we are trying to provide more machines in other data centers to reduce the impact of the issue.

Thank you for your patience!

You can check for updates here:
https://status.codemagic.io/

[salimazeriDev]

Same here, we had that problems on Friday, and today we have same situation.

[mrruby]

looks like its working for me now

[salimazeriDev]

Yeah, I see that my builds passed that failing step. Hope, it's not just a luck as I have few builds to make today.

[emanuel-braz]

not for me 😔

[burekas7]

Now it's working for me too.
But the whole build process takes much longer then usual.

[sobe-wt]

Happening for me now ;(

[warpaltarpers]

Seems like, as stated by @mikhail-tokarev before, the increased load during peak work hours can bring the issue out

[mikhail-tokarev]

First of all, I’m sorry for the long silence in this thread. I'd like to provide an update on the current status and our next steps to resolve the problem. I want to assure all of you that we are taking the situation very seriously and doing everything possible to address it. I'll be happy to answer any questions you may have in the comments.

As you are aware, we are still experiencing significant challenges with network performance. Currently, the data center has identified issues with the hardware of the firewalls, caused by high traffic. These issues are causing a high packet drop ratio, resulting in slow download speeds and timeouts.

• We did not receive any notifications of the traffic increase beforehand, leaving us unprepared to address the issue proactively.
• Furthermore, we are unable to quickly upgrade the firewalls or add more machines to another data center, because our partner has encountered issues with the supply chain.

We weren’t prepared for such a scenario, and we completely understand that we have performed poorly, putting you in a situation where you cannot rely on our service anymore.

Currently, we are planning the following steps to mitigate the issue:

1. As a workaround, the data center will setup backup firewalls to create a new subnet to reduce the firewall load (UPDATE: Maintenance Mar 28 at 02:00pm UTC – Mar 28 at 05:00pm UTC).
2. We expect the first machines to arrive later this week to help distribute our load.
3. We are also exploring alternative solutions to prevent such incidents in the future.

I sincerely appologize for the troubles caused!

[thassio-vinicius]

Are the workarounds already in place? If not, any ETA? I tried making a new build just now and it's still failing

[mikhail-tokarev]

unfortunately, not yet. I'll update the thread once we know anything from the data center engineers.

[mikhail-tokarev]

At this moment, we still haven't received any information or confirmation from the data center engineers. We expect emergency maintenance tomorrow. I apologize for yet another delay.

[Imgkl]

Thanks @mikhail-tokarev for the update and the action plan.

FWIW the builds are in queue for the last 30 mins.

[mikhail-tokarev]

thanks, we are currently restoring machines to the pool as they lost network connection (almost simultaneously). We haven't received any announcement and not sure if the data center is making any changes or not. However all builds should start soon.

[helinanever]

There is going to be emergency maintenance starting at 14:00 UTC today with an estimated duration of 3 hours.

Our data center partner will move the Apple M1 machines (mac_mini_m1) to a new network to improve network performance.

During the maintenance window, we will be operating with a reduced number of M1 build machines. As a result, you may be experiencing longer than usual build queues.

Apple M2 and Mac Pro, Linux and Windows machine availability will not be affected.

Thank you for your understanding.

Please keep an eye on the status page for updates: https://status.codemagic.io/

[mikhail-tokarev]

It still takes quite some time to resolve the build queue backlog.

[mikhail-tokarev]

Currently, all services are operating as expected. We continue to monitor the situation closely and are working with data center engineers to improve network stability.

[mikhail-tokarev]

There is another ongoing emergency maintenance, so we will be operating with a reduced number of M1 build machines. As a result, you may experience longer than usual build queues.

Please keep an eye on the status page for updates: https://status.codemagic.io/

[mikhail-tokarev]

We need to extend the maintenance period. We apologize for any inconvenience this may cause.

[mikhail-tokarev]

The maintenance has been completed and all builds will start soon.

[ChrisElliotUK]

Thanks Mikhail. I really appreciate the update and the updates on GitHub. The communication has been very good so massive applause to your team. Christopher ElliotOn 28 Mar 2024, at 19:14, Mikhail Tokarev ***@***.***> wrote: The data center engineers have completed the new subnet configuration. We are now awaiting an update on the ETA for when the machines will be physically switched to the new network. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[irby]

Thank you for your hard work on this, guys!

[bastienlemaitre]

Any update ?

[mikhail-tokarev]

sorry, just posted update here https://github.com/orgs/codemagic-ci-cd/discussions/2424#discussioncomment-8940664

[eduardoazvd17]

Its still not working for me, im having timeout issues on installing dependencies section. I haven't faced it before.

[eduardoazvd17]

Of course, ID: 6606c5f9f13b41ec5f927dff

[eduardoazvd17]

I tryed start a new build but its keep queued: 6606d73cde39e009d29e17cf

[mikhail-tokarev]

thanks for sharing build id. I'll try to check time issue with time out
re queued builds I've post update here https://github.com/orgs/codemagic-ci-cd/discussions/2424#discussioncomment-8953207

[mikhail-tokarev]

The machine was disconnected from the network while running your build 6606c5f9f13b41ec5f927dff. Currently, we are awaiting any confirmation from the data center engineers. However, it seems they started reconfiguring hosts without letting us know, so we didn't have a chance to remove hosts from the pool :(

[eduardoazvd17]

Ok, thanks for reply. My build was completed successfully now.

[momer]

I'm also encountering a stuck, queued, build at build id: 6606dada350295535e82475b

[mikhail-tokarev]

check update here https://github.com/orgs/codemagic-ci-cd/discussions/2424#discussioncomment-8953207

[jhonnyoddball]

My build (6606c9400f14a76b72028590) took an hour and never finished. I had to cancel it.
Started a new build (6606da07de39e009d29e18db) has been queued for more than 30 mins now

[mikhail-tokarev]

I've posted update here https://github.com/orgs/codemagic-ci-cd/discussions/2424#discussioncomment-8953207

[mikhail-tokarev]

🔔 Important Announcement 🔔

We sincerely apologize for the recent network issues that have impacted many of you. We understand the frustration that comes with being unable to use CI/CD and release new versions, and we want to assure you that ensuring service reliability is our top priority.

As a gesture of our apology and appreciation for your patience during this time, we're offering you a 50% discount on your next payment. Simply use the code WEARESORRY in your team billing settings to redeem your discount.

We are committed to providing you with the best possible experience. If you have any further questions or concerns, please don't hesitate to reach out to me or our customer support team.

[ggirotto]

@mikhail-tokarev How to get this discount for those in annual billing plans?

[mikhail-tokarev]

Hey @ggirotto ! please ask the support team using in-app chat widget

[dtrdic]

Just a note for users who don't know how to apply the coupon. The promotion code can be applied in the billing team settings under the billing overview section.

[himesh-cm]

While we are closing our discussion for now, if you have any other questions or suggestions, feel free to reply back or start a new discussion anytime. Have a nice day!

[matmicro]

Facing same issue currently.
Any ETA to get this resolved ?

[mikhail-tokarev]

hey, this issue has beed resolved long time ago. You can try to restart your build and if issue persists, open a new thread, and provide build id and more details please. Someone from the team will take a look. Thank you!