Does not work with Docker >= 25?

[romaincabassot]

Hello,

We use your clair-scanner since some time now and it worked well but recently we updated our Docker version and I cannot manage to get it working again.

On a Debian 12 server it failed with latest Docker engine version (26.1.3), I tried to downgrade Docker because I wasn't able to identify an issue on our side: it doesn't work with 25.0.5 but works with 24.0.9.
As this server is configured with certificates, swarm, docker specific options, etc I tried the same on my Laptop (Ubuntu 22.04) with Docker engine version 26.1.3 and no specific configuration. It fails on 26.1.3, works on 24.0.9.

My local test:

# Download static binary https://github.com/arminc/clair-scanner/releases/download/v12/clair-scanner_linux_amd64
docker run -d --name clair-db arminc/clair-db:latest
docker run -p 6060:6060 --link clair-db:postgres -d --name clair arminc/clair-local-scan:latest
docker pull nginx:latest
./clair-scanner -c http://10.3.101.223:6060 --ip=10.3.101.223 nginx:latest

2024/06/04 17:14:54 [INFO] ▶ Start clair-scanner
2024/06/04 17:14:55 [INFO] ▶ Server listening on port 9279
2024/06/04 17:14:55 [INFO] ▶ Analyzing blobs/sha256/5d4427064ecc46e3c2add169e9b5eafc7ed2be7861081ec925938ab628ac0e25
2024/06/04 17:14:55 [CRIT] ▶ Could not analyze layer: Clair responded with a failure: Got response 400 with message {"Error":{"Message":"could not find layer"}}

Clair logs:

docker logs clair
{"Event":"running database migrations","Level":"info","Location":"pgsql.go:216","Time":"2024-06-04 15:14:48.411963"}
{"Event":"database migration ran successfully","Level":"info","Location":"pgsql.go:223","Time":"2024-06-04 15:14:48.566078"}
{"Event":"starting main API","Level":"info","Location":"api.go:52","Time":"2024-06-04 15:14:48.566141","port":6060}
{"Event":"notifier service is disabled","Level":"info","Location":"notifier.go:77","Time":"2024-06-04 15:14:48.566231"}
{"Event":"starting health API","Level":"info","Location":"api.go:85","Time":"2024-06-04 15:14:48.566224","port":6061}
{"Event":"updater service started","Level":"info","Location":"updater.go:83","Time":"2024-06-04 15:14:48.566260","lock identifier":"b8e9ff31-e39d-49ca-ad43-22a46a4b31c0"}
{"Event":"updating vulnerabilities","Level":"info","Location":"updater.go:192","Time":"2024-06-04 15:14:48.663086"}
{"Event":"fetching vulnerability updates","Level":"info","Location":"updater.go:239","Time":"2024-06-04 15:14:48.663143"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:85","Time":"2024-06-04 15:14:48.663159","package":"Ubuntu"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"amzn.go:84","Time":"2024-06-04 15:14:48.663170","package":"Amazon Linux 2"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"oracle.go:119","Time":"2024-06-04 15:14:48.663182","package":"Oracle Linux"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"alpine.go:176","Time":"2024-06-04 15:14:48.663262","package":"Alpine"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"rhel.go:94","Time":"2024-06-04 15:14:48.663282","package":"RHEL"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"amzn.go:84","Time":"2024-06-04 15:14:48.663354","package":"Amazon Linux 2018.03"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"debian.go:63","Time":"2024-06-04 15:14:48.663442","package":"Debian"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2024-06-04 15:14:49.596308","updater name":"amzn2"}
{"Event":"Debian trixie is not mapped to any version number (eg. Jessie-\u003e8). Please update me.","Level":"warning","Location":"debian.go:134","Time":"2024-06-04 15:14:49.712295"}
{"Event":"Debian bookworm is not mapped to any version number (eg. Jessie-\u003e8). Please update me.","Level":"warning","Location":"debian.go:134","Time":"2024-06-04 15:14:49.712333"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2024-06-04 15:14:49.712347","updater name":"debian"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2024-06-04 15:14:50.275997","updater name":"alpine"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2024-06-04 15:14:50.895782","updater name":"rhel"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2024-06-04 15:14:51.126010","updater name":"amzn1"}
{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2024-06-04 15:14:55.339157","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2024-06-04 15:14:55.339289","error":"could not find layer","layer":"blobs/sha256/5d4427064ecc46e3c2add169e9b5eafc7ed2be7861081ec925938ab628ac0e25","path":"http://10.3.101.223:9279/blobs/sha256/5d4427064ecc46e3c2add169e9b5eafc7ed2be7861081ec925938ab628ac0e25/layer.tar"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2024-06-04 15:14:55.340096","elapsed time":93638158,"method":"POST","remote addr":"10.3.101.223:38028","request uri":"/v1/layers","status":"400"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2024-06-04 15:15:35.639606","updater name":"oracle"}

Is the issue known?

Thanks
Romain

[romaincabassot]

After some digging, it seems that Docker >24 saves Docker images in OCI format (moby/moby#44598).

This change is incompatible with the old Clair version bundled in clair-local-scanner (tar filetree has changed, manifest has changed, ...).
I haven't found a way in Docker to fallback to legacy Docker images format but some tools are able to do it (Skopeo or regclient for example).

[cpuguy83]

The actual layout in the tar changed, however the manifest did not and the manifest, prior to the mentioned change, is the source of truth for how to read the tarball.
Now either the older docker-style manifest or the OCI index can be used to parse it.

[romaincabassot]

Thanks for the comments @cpuguy83

@arminc I just made a PR to fix this

[arminc]

@romaincabassot Thanks for the PR and the fix.

@datadot Is this something you want to look at?

[datadot]

@arminc thanks taking a look.
@romaincabassot I have been toying with a fix that is a wider fix for a couple weeks, ie upgrading CLAIR, rewrite of this tool to work with that, however I've been hitting a brick wall with the new CLAIR container being 25Gb! I'll take a look at this PR as it could be a reasonable stop gap whist I work through solution to the size!