Flatcar machines stuck in Waiting... instead of pulling new release(s)

[tylerauerbeck]

Description

After flipping pin to latest stable, a number of machines pulled down the latest download. We had paused downloads reboots over the weekend and came back to begin again a few days later and now we have a number of machines that are stuck in Current Status of Waiting.... When looking at update_engine logs the only mention I see is omaha_request_action.cc:629] HTTP reported success but Omaha reports an error.. When lining this up for a similar error in the Nebraska logs matching that machineID, the only log I see is update complete.error. Is there any additional logging I can turn up to determine the actual root of this problem. I've tried things like restarting update_engine, Nebraska, etc. to see if I can get things unstuck without any luck.

Impact

Further downloads are not occurring and current_status is not accurately reflecting the status of this rollout.

Environment and steps to reproduce

1. Set-up: Nebraska 2.9 attempting to roll out 3975.2.1
2. Task: Flipped channel pin to 3975.2.1 to begin rollout
3. Action(s): Update pin to begin rollout, paused update of machines over a span of 2+ days (resulting in machines staying in the Downloaded state for a period of time prior to attempting to continue rollout
   a. [ requested the start of a new pod or container ]
   b. [ container image downloaded ]
4. Error: [describe the error that was triggered]

• omaha_request_action.cc:629] HTTP reported success but Omaha reports an error.
• update complete.error

Expected behavior

Nebraska accurately reflecting current status and additional nodes continuing to download new release

Additional information

N/A

[tylerauerbeck]

If it helps at all, the nodes in Waiting... seem to bunch up under On Hold in the dashboard for the channel.