Corrupted cache #16

joeybrown-sf · 2024-07-02T20:38:39Z

Summary

LOW-2: Denial-of-Service (DoS) provoked by removing build cache tarballs or altering the OCI image manifest

This PR addresses the security concern by skipping the cache restore if the tarball is not present.

A user will see the a similar warning as the following. The restore no longer hard fails.

Warning: Layer with SHA 'sha256:5a1185b2a8f6e19dfbd7db44d1cd31e67873798bc2a38eb3c3df5bba733d0eb4' not found, skipping restore

Release notes

Restorer no longer fails when missing cache artifacts are referenced in the cache metadata.

Context

Signed-off-by: Natalie Arellano <[email protected]>

…ot, (#1348) consider it not a match Signed-off-by: Natalie Arellano <[email protected]>

… labels (#1347) * Read `/etc/os-release` file when distro information is not present in labels Signed-off-by: Pavel Busko <[email protected]> * fix unit tests Signed-off-by: Pavel Busko <[email protected]> * Update phase/generator_test.go Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Pavel Busko <[email protected]> * Update phase/generator_test.go Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Pavel Busko <[email protected]> --------- Signed-off-by: Pavel Busko <[email protected]> Signed-off-by: Pavel Busko <[email protected]> Co-authored-by: Natalie Arellano <[email protected]>

Bumps the go-dependencies group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/GoogleContainerTools/kaniko](https://github.com/GoogleContainerTools/kaniko) | `1.21.1` | `1.22.0` | | [github.com/containerd/containerd](https://github.com/containerd/containerd) | `1.7.14` | `1.7.16` | | [github.com/docker/docker](https://github.com/docker/docker) | `26.0.1+incompatible` | `26.1.0+incompatible` | | [github.com/moby/buildkit](https://github.com/moby/buildkit) | `0.13.1` | `0.13.2` | | [golang.org/x/sys](https://github.com/golang/sys) | `0.18.0` | `0.19.0` | Updates `github.com/GoogleContainerTools/kaniko` from 1.21.1 to 1.22.0 - [Release notes](https://github.com/GoogleContainerTools/kaniko/releases) - [Changelog](https://github.com/GoogleContainerTools/kaniko/blob/main/CHANGELOG.md) - [Commits](GoogleContainerTools/kaniko@v1.21.1...v1.22.0) Updates `github.com/containerd/containerd` from 1.7.14 to 1.7.16 - [Release notes](https://github.com/containerd/containerd/releases) - [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md) - [Commits](containerd/containerd@v1.7.14...v1.7.16) Updates `github.com/docker/docker` from 26.0.1+incompatible to 26.1.0+incompatible - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.0.1...v26.1.0) Updates `github.com/moby/buildkit` from 0.13.1 to 0.13.2 - [Release notes](https://github.com/moby/buildkit/releases) - [Commits](moby/buildkit@v0.13.1...v0.13.2) Updates `golang.org/x/sys` from 0.18.0 to 0.19.0 - [Commits](golang/sys@v0.18.0...v0.19.0) --- updated-dependencies: - dependency-name: github.com/GoogleContainerTools/kaniko dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/containerd/containerd dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/moby/buildkit dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Natalie Arellano <[email protected]>

Bumps the go-dependencies group with 2 updates: [github.com/docker/docker](https://github.com/docker/docker) and [golang.org/x/sys](https://github.com/golang/sys). Updates `github.com/docker/docker` from 26.1.0+incompatible to 26.1.1+incompatible - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.1.0...v26.1.1) Updates `golang.org/x/sys` from 0.19.0 to 0.20.0 - [Commits](golang/sys@v0.19.0...v0.20.0) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…t compat (#1352) * Reorder functions in file Signed-off-by: Natalie Arellano <[email protected]> * Also read distro information from /etc/os-release when checking target compat buildpacks/lifecycle#1347 reads the file when providing target env vars to buildpacks during detect, but we also need to consider this info when deciding whether or not to run detect for the buildpack Signed-off-by: Natalie Arellano <[email protected]> * Error if we don't find run image OS during analyze And remove checks for missing OS later in the build, as it should always be there Signed-off-by: Natalie Arellano <[email protected]> --------- Signed-off-by: Natalie Arellano <[email protected]>

* fix: instance create issue for GH actions Signed-off-by: Dilip Gowda Bhagavan <[email protected]> * fix: ibm cloud instance creation issue Signed-off-by: Dilip Gowda Bhagavan <[email protected]> --------- Signed-off-by: Dilip Gowda Bhagavan <[email protected]>

* More fixes for target compat checking during detect - If a buildpack fails to specify os/arch (but specifies distro) still check targets - If the run image fails to specify os/arch (this should not happen actually as we will fail during analyze) still check targets - Fix typo in buildpack descriptor struct so that we actually get stack information - If we get distro information from /etc/os-release, persist this information to later invocations to that the log message printed when errors are encountered will be accurate - Don't override inner `i` in loop (this should not actually affect the outer loop but is confusing) Signed-off-by: Natalie Arellano <[email protected]> * When stack is "any", don't infer empty target as it is not needed Missing targets is sufficient for wildcard match Signed-off-by: Natalie Arellano <[email protected]> * Remove backwards compatible glue that actually causes fewer builds to succeed Fixes buildpacks/lifecycle#1355 Signed-off-by: Natalie Arellano <[email protected]> * Remove exit (this was added for debugging purposes) Signed-off-by: Natalie Arellano <[email protected]> --------- Signed-off-by: Natalie Arellano <[email protected]>

Bumps [azure/docker-login](https://github.com/azure/docker-login) from 1 to 2. - [Release notes](https://github.com/azure/docker-login/releases) - [Commits](Azure/docker-login@v1...v2) --- updated-dependencies: - dependency-name: azure/docker-login dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Natalie Arellano <[email protected]>

* Bump the go-dependencies group across 1 directory with 5 updates Bumps the go-dependencies group with 3 updates in the / directory: [github.com/BurntSushi/toml](https://github.com/BurntSushi/toml), [github.com/GoogleContainerTools/kaniko](https://github.com/GoogleContainerTools/kaniko) and [github.com/containerd/containerd](https://github.com/containerd/containerd). Updates `github.com/BurntSushi/toml` from 1.3.2 to 1.4.0 - [Release notes](https://github.com/BurntSushi/toml/releases) - [Commits](BurntSushi/toml@v1.3.2...v1.4.0) Updates `github.com/GoogleContainerTools/kaniko` from 1.22.0 to 1.23.0 - [Release notes](https://github.com/GoogleContainerTools/kaniko/releases) - [Changelog](https://github.com/GoogleContainerTools/kaniko/blob/main/CHANGELOG.md) - [Commits](GoogleContainerTools/kaniko@v1.22.0...v1.23.0) Updates `github.com/awslabs/amazon-ecr-credential-helper/ecr-login` from 0.0.0-20231213181459-b0fcec718dc6 to 0.0.0-20240419161514-af205d85bb44 - [Release notes](https://github.com/awslabs/amazon-ecr-credential-helper/releases) - [Changelog](https://github.com/awslabs/amazon-ecr-credential-helper/blob/main/CHANGELOG.md) - [Commits](https://github.com/awslabs/amazon-ecr-credential-helper/commits) Updates `github.com/containerd/containerd` from 1.7.16 to 1.7.17 - [Release notes](https://github.com/containerd/containerd/releases) - [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md) - [Commits](containerd/containerd@v1.7.16...v1.7.17) Updates `github.com/docker/docker` from 26.1.1+incompatible to 26.1.2+incompatible - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.1.1...v26.1.2) --- updated-dependencies: - dependency-name: github.com/BurntSushi/toml dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/GoogleContainerTools/kaniko dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/awslabs/amazon-ecr-credential-helper/ecr-login dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/containerd/containerd dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> * Fix unit Signed-off-by: Natalie Arellano <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Natalie Arellano <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Natalie Arellano <[email protected]>

Signed-off-by: Joey Brown <[email protected]>

* Ensure read access to the run image selected by extensions Co-authored-by: Nicolas Bender <[email protected]> Signed-off-by: Pavel Busko <[email protected]> Co-authored-by: Pavel Busko <[email protected]> * move read access check to the restorer cmd Signed-off-by: Pavel Busko <[email protected]> * guard behind platform version check Signed-off-by: Pavel Busko <[email protected]> --------- Signed-off-by: Pavel Busko <[email protected]> Co-authored-by: Nicolas Bender <[email protected]>

* Restore cached launch layers not found in appLayers Co-authored-by: Ralf Pannemans <[email protected]> Signed-off-by: Pavel Busko <[email protected]> * add platform api guard Signed-off-by: Pavel Busko <[email protected]> --------- Signed-off-by: Pavel Busko <[email protected]> Co-authored-by: Ralf Pannemans <[email protected]>

phase/restorer.go

phase/cache.go

phase/exporter.go

cache/image_cache.go

cache/volume_cache.go

cache/image_cache.go

cache/volume_cache.go

natalieparellano · 2024-07-08T17:09:56Z

Thank you for this @joeybrown-sf! I left a few nits and a few questions

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

Signed-off-by: Joey Brown <[email protected]>

* Target data: populate os/arch as well as distro information Fixes buildpacks/lifecycle#1371 Signed-off-by: Natalie Arellano <[email protected]> * Only os and arch are truly required Signed-off-by: Natalie Arellano <[email protected]> * Fix one more unit Signed-off-by: Natalie Arellano <[email protected]> * Fix even more units Signed-off-by: Natalie Arellano <[email protected]> --------- Signed-off-by: Natalie Arellano <[email protected]>

* Bump the go-dependencies group across 1 directory with 6 updates Bumps the go-dependencies group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/GoogleContainerTools/kaniko](https://github.com/GoogleContainerTools/kaniko) | `1.23.0` | `1.23.1` | | [github.com/containerd/containerd](https://github.com/containerd/containerd) | `1.7.17` | `1.7.19` | | [github.com/google/go-containerregistry](https://github.com/google/go-containerregistry) | `0.19.1` | `0.20.0` | | [github.com/moby/buildkit](https://github.com/moby/buildkit) | `0.13.2` | `0.14.1` | | [golang.org/x/sys](https://github.com/golang/sys) | `0.20.0` | `0.22.0` | Updates `github.com/GoogleContainerTools/kaniko` from 1.23.0 to 1.23.1 - [Release notes](https://github.com/GoogleContainerTools/kaniko/releases) - [Changelog](https://github.com/GoogleContainerTools/kaniko/blob/main/CHANGELOG.md) - [Commits](GoogleContainerTools/kaniko@v1.23.0...v1.23.1) Updates `github.com/containerd/containerd` from 1.7.17 to 1.7.19 - [Release notes](https://github.com/containerd/containerd/releases) - [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md) - [Commits](containerd/containerd@v1.7.17...v1.7.19) Updates `github.com/docker/docker` from 26.1.2+incompatible to 26.1.3+incompatible - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.1.2...v26.1.3) Updates `github.com/google/go-containerregistry` from 0.19.1 to 0.20.0 - [Release notes](https://github.com/google/go-containerregistry/releases) - [Changelog](https://github.com/google/go-containerregistry/blob/main/.goreleaser.yml) - [Commits](google/go-containerregistry@v0.19.1...v0.20.0) Updates `github.com/moby/buildkit` from 0.13.2 to 0.14.1 - [Release notes](https://github.com/moby/buildkit/releases) - [Commits](moby/buildkit@v0.13.2...v0.14.1) Updates `golang.org/x/sys` from 0.20.0 to 0.22.0 - [Commits](golang/sys@v0.20.0...v0.22.0) --- updated-dependencies: - dependency-name: github.com/GoogleContainerTools/kaniko dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/containerd/containerd dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/google/go-containerregistry dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/moby/buildkit dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> * Stay on the same version of buildkit as kaniko Signed-off-by: Natalie Arellano <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Natalie Arellano <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Natalie Arellano <[email protected]>

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.64.0 to 1.64.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.64.0...v1.64.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Surface registry error instead of logging it as debug When permissions issues are encountered, it can be hard to determine the root cause without the error returned from the registry Signed-off-by: Natalie Arellano <[email protected]> * Fix test expectation Signed-off-by: Natalie Arellano <[email protected]> --------- Signed-off-by: Natalie Arellano <[email protected]>

Image utils should fail with a Layer Not found in both ReuseLayer & GetLayer. For GetLayer, when there is a missing blob, it's return an unexpected EOF error. For ReuseLayer, when there is a missing blob, it's not returning an error but it should.

cache/volume_cache.go

natalieparellano · 2024-07-16T17:10:22Z

Looks like these units are failing:

=== CONT  TestImageCache/ImageCache/#Commit/with_#ReuseLayer/reuse_without_commit/retrieve_returns_the_previous_layer
    image_cache_test.go:278: Expected nil: failed to reuse cache layer with SHA 'sha256:797bb0abff798d7200af7685dca7901edffc52bf2[650](https://github.com/buildpacks/lifecycle-private/actions/runs/9958919901/job/27514455016?pr=16#step:6:651)0d5bd97282658ee24152'
=== CONT  TestImageCache/ImageCache/#Commit/with_#ReuseLayer/reuse_then_commit/returns_the_reused_layer
    image_cache_test.go:252: Expected nil: failed to reuse cache layer with SHA 'sha256:797bb0abff798d7200af7685dca7901edffc52bf26500d5bd97282658ee24152'

joeybrown-sf · 2024-07-16T19:22:17Z

oops I introduced a regression. Thanks for that call out!

I think we're good now. Please take a look at your convenience 😃

natalieparellano · 2024-07-16T20:49:35Z

Thank you for the PR @joeybrown-sf! This looks good to me.

natalieparellano and others added 12 commits April 22, 2024 15:06

Bump imgutil to pick up fix from buildpacks/imgutil#269 (#1343)

37d6985

Signed-off-by: Natalie Arellano <[email protected]>

Try to fix test-s390x workflow (#1339)

13d3691

Signed-off-by: Natalie Arellano <[email protected]>

When a buildpack declares distro information, but a base image does n…

44b7041

…ot, (#1348) consider it not a match Signed-off-by: Natalie Arellano <[email protected]>

Bump imgutil to pick up fixes for containerd and podman (#1361)

36c0af0

Signed-off-by: Natalie Arellano <[email protected]>

joeybrown-sf requested a review from a team as a code owner July 2, 2024 20:38

add failing test to restorer

9e19760

Signed-off-by: Joey Brown <[email protected]>

joeybrown-sf force-pushed the corrupted-cache branch from 60fad2c to e505545 Compare July 2, 2024 20:40

joeybrown-sf added 2 commits July 2, 2024 15:41

restorer and exporter working as expected

376931a

Signed-off-by: Joey Brown <[email protected]>

lint

1c4aefa

Signed-off-by: Joey Brown <[email protected]>

joeybrown-sf force-pushed the corrupted-cache branch from e505545 to 1c4aefa Compare July 2, 2024 20:45

pbusko and others added 2 commits July 3, 2024 10:01

natalieparellano reviewed Jul 8, 2024

View reviewed changes

cache/volume_cache.go Outdated Show resolved Hide resolved

joeybrown-sf and others added 7 commits July 9, 2024 10:10

Update phase/restorer.go

0101e33

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

Update phase/cache.go

3d30c91

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

Update cache/image_cache.go

8570898

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

Update cache/volume_cache.go

2c58c8d

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

Update cache/volume_cache.go

55aa264

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

Update cache/volume_cache.go

de0806a

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

Update cache/volume_cache.go

1c304d5

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

joeybrown-sf and others added 9 commits July 9, 2024 11:13

Update cache/volume_cache.go

80d089f

Co-authored-by: Natalie Arellano <[email protected]> Signed-off-by: Joey Brown <[email protected]>

update based on feedback

6a2a590

Signed-off-by: Joey Brown <[email protected]>

fix log

dcea1b3

temp fix

dba6b9a

joeybrown-sf force-pushed the corrupted-cache branch from c9516cd to 67a8cf5 Compare July 12, 2024 16:13

natalieparellano reviewed Jul 15, 2024

View reviewed changes

cache/volume_cache.go Show resolved Hide resolved

natalieparellano reviewed Jul 15, 2024

View reviewed changes

cache/volume_cache.go Show resolved Hide resolved

joeybrown-sf added 2 commits July 15, 2024 16:29

add eof check

1b5aa7c

add not exist check

6a779e3

joeybrown-sf added 2 commits July 16, 2024 14:04

reuse layer test

ba02977

fix test regression

289442a

joeybrown-sf requested a review from natalieparellano July 16, 2024 19:22

natalieparellano approved these changes Jul 16, 2024

View reviewed changes

jabrown85 approved these changes Jul 16, 2024

View reviewed changes

Merge branch 'main' into corrupted-cache

39f37e7

natalieparellano mentioned this pull request Jul 17, 2024

Security review: recover corrupt caches buildpacks/lifecycle#1382

Open

joeybrown-sf merged commit 91f791c into main Jul 17, 2024
1 of 5 checks passed

natalieparellano mentioned this pull request Oct 22, 2024

Recover corrupted volume cache buildpacks/lifecycle#1410

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Corrupted cache #16

Corrupted cache #16

joeybrown-sf commented Jul 2, 2024

natalieparellano commented Jul 8, 2024

natalieparellano commented Jul 16, 2024

joeybrown-sf commented Jul 16, 2024

natalieparellano commented Jul 16, 2024

Corrupted cache #16

Corrupted cache #16

Conversation

joeybrown-sf commented Jul 2, 2024

Summary

Release notes

Related

Context

natalieparellano commented Jul 8, 2024

natalieparellano commented Jul 16, 2024

joeybrown-sf commented Jul 16, 2024

natalieparellano commented Jul 16, 2024