for update-stemcell, only download releases if necessary

[dlinsley]

implements #492

• Only download releases if determining the SHA1 remotely was not possible.
• Updates artifactory GetMatchedRelease to include remote SHA1 of matched release from remote
• Updates bosh.io GetMatchedRelease to include remote SHA1 of matched release from remote
• Other release sources do not appear to support determining the remote SHA1.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[drich10]

Change looks good and I agree that defaulting this behavior rather than requiring a flag is correct.

Afaict there isn't a test that verifies the behavior change in update_stemcell.go itself. If I'm right, would you mind adding one?

[crhntr]

Where is "not-calculated" from? Maybe pull this out into a const and add a note on what emits this value. If there was a test, that would also solve this question.

(It was cool to see the PR review request. I miss y'all. Hope you're all well.)

[dlinsley]

The s3 and github sources can set this as not-calculated particularly when no-download specified.

[crhntr]

I added a comment. I don't recall anything that would break from this change.

This issue from 2021 is kinda related. We only calculate the SHA1 because we used to have a huge s3 bucket that did not have object checksums enabled and adding it would have been super computationally expensive. Funnily, if we had enabled that feature we would have saved thousands of release downloads.

#297

[dlinsley]

Change looks good and I agree that defaulting this behavior rather than requiring a flag is correct.

Afaict there isn't a test that verifies the behavior change in update_stemcell.go itself. If I'm right, would you mind adding one?

The updated tests add a third release to the tile under test that provides sufficient metadata to the update-stemcell method to not downloaded the release. Updated the returned metadata of the existing releases in the tile to explicitly return either empty string or not-calculated to trigger a calls to download the releases.

[drich10]

Appreciate the updates and work! Approving and merging 🙏

[rizwanreza]

This broke our pipelines. It is not downloading the releases in our compilation manager.

I am going to revert this PR and re-release kiln.

[dlinsley]

Sorry about that, didn't realize the download was desired for this command, may put this behind a --no-download flag instead