Use OCI manifest from skopeo copy dir for analysis

[tsaarni]

If uncompressed layers are stored to the registry, skopeo will compress them on the fly when executing "skopeo copy ... oci:dir". In this case there will be disparity between the image manifest that was previously fetched with "skopeo inspect", which had references to uncompressed layers. Analysis will look up for uncompressed layers from the OCI directory and raise an exception since they cannot be found - analysis will fail.

This change makes the analysis use the OCI image manifest and therefore resolve to the correct filenames.

Fixes #1080

Signed-off-by: Tero Saarni [email protected]

[zhill]

Thanks for submitting this! I'd also like to copy the image you used into our testing repo so we can use it for regression tests in the future as well. I'll look into it, but if you used a specific process to upload it I would appreciate that info so we can use it was well.

[tsaarni]

Thanks for submitting this! I'd also like to copy the image you used into our testing repo so we can use it for regression tests in the future as well. I'll look into it, but if you used a specific process to upload it I would appreciate that info so we can use it was well.

I manually uncompressed a layer from alpine, modified its manifest and then used the registry REST API directly from command line to upload, since skopeo always would compress the layers. But this is so convoluted, I bet there must be some straightforward way to trigger this as well. I'll look into it and will come back.

By the way, I had some doubts when removing the manifest and parent_manifest from the retrying_pull_image() parameters. It seemed to me like they are not necessary anymore and maybe they were only needed before moving to OCI directory layout for the work dir. If that is true, then maybe some more cleanups can be done in skopeo_wrapper.py:download_image().

[tsaarni]

I've found out that the problematic images that lead to submitting #1080 were uploaded by proprietary software which directly interacts with the registry REST API. So far, I don't have any indication that there exists any publicly available tool that would result in similar images.

I've created a simple script that automates my manual upload steps: upload-as-uncompressed.py

It is very bare bones, no error handling etc but might be ok for demonstration purposes...

[Vijay-P]

@tsaarni thanks for submitting this PR. It looks like Circle CI missed this PR for some reason. Can you rebase on master or merge master into your branch? Hopefully that will get a new CI job to run.

[tsaarni]

@Vijay-P Sure, I've now rebased the PR.

[Vijay-P]

@Vijay-P Sure, I've now rebased the PR.

Thanks. Not sure what the issue is, but creating a temporary duplicate branch and PR triggered the CI workflow.

[zhill]

I think this will have an impact on images that use a foreign.diff layer type. The manifest is passed in for that computation, so that the layers can be fetched. But, since containers/image#1063 was merged, we may not need that code at all anymore if an appropriate update to skopeo was done as well. It doesn't look like it was, but will confirm.

[wimi]

This PR seems to be pretty old, but problem seems to be still present. I've just tested this change on my custom build and it allowed me to scan images that were previously failing (we are not authors of these - they are actually from Microsoft).

Can we merge this?