Trivy randomly fail to find vulnerabilities and reports 0 issues

[krab-skunk]

Description

We run a Trivy scanning job on an AWS EC2, and sometimes it will find every issues for all images, and few hours later, for the very same scan with the same images, it will simply find 0 vulnerabilities for all of the images, while the previous scan found 13000+ CVEs

Here are an example log: example of scanning bitnami/redis-exporter:1.43.0-debian-11-r4 and report 0 issues

trivy image bitnami/redis-exporter:1.43.0-debian-11-r4 --scanners vuln --debug
2024-10-17T17:59:21Z	DEBUG	No plugins loaded
2024-10-17T17:59:21Z	DEBUG	Default config file "file_path=trivy.yaml" not found, using built in values
2024-10-17T17:59:21Z	DEBUG	Cache dir	dir="/home/app/.cache/trivy"
2024-10-17T17:59:21Z	DEBUG	Cache dir	dir="/home/app/.cache/trivy"
2024-10-17T17:59:21Z	DEBUG	Parsed severities	severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-10-17T17:59:21Z	DEBUG	Ignore statuses	statuses=[]
2024-10-17T17:59:21Z	DEBUG	DB update was skipped because the local DB is the latest
2024-10-17T17:59:21Z	DEBUG	DB info	schema=2 updated_at=2024-10-17T12:17:37.660183081Z next_update=2024-10-18T12:17:37.660182941Z downloaded_at=2024-10-17T17:48:32.650403883Z
2024-10-17T17:59:24Z	DEBUG	[pkg] Package types	types=[os library]
2024-10-17T17:59:24Z	DEBUG	[pkg] Package relationships	relationships=[unknown root direct indirect]
2024-10-17T17:59:24Z	INFO	[vuln] Vulnerability scanning is enabled
2024-10-17T17:59:24Z	DEBUG	Enabling misconfiguration scanners	scanners=[azure-arm cloudformation dockerfile helm kubernetes terraform terraformplan-json terraformplan-snapshot]
2024-10-17T17:59:24Z	DEBUG	Initializing scan cache...	type="fs"
2024-10-17T17:59:24Z	DEBUG	[image] Detected image ID	image_id="sha256:6a6d68284420f8ec7f3864464de8cf59510968954d22b1f0f4beb27ffa507966"
2024-10-17T17:59:24Z	DEBUG	[image] Detected diff ID	diff_ids=[sha256:9b35e8d8e8bf952b054748f36350bd3c6c3acd3c157d242b38019f90eb0a6a54 sha256:c667a923a11722e8e9fdd30675519973ec632dfbae7bf33d310004c0d552c215 sha256:9c8e78724ec633b64c0c0d5880ae00f4e838e3a343248799165c797e3daf9d04 sha256:1fe04252c67e8a5c9f403abc8639041f01e3a337b431ab9b55442b408e9905b4 sha256:00d41617810c06cbb85a472749e8da049c11a9877599a6518e929a33c4a0cb33 sha256:057558b45c7fea60a0a753389289dcd678892c0cb0dd1a538f5c593d12dd2ed0]
2024-10-17T17:59:24Z	DEBUG	[image] Detected base layers	diff_ids=[]
2024-10-17T17:59:24Z	INFO	Detected OS	family="debian" version="11.3"
2024-10-17T17:59:24Z	INFO	[debian] Detecting vulnerabilities...	os_version="11" pkg_num=110
2024-10-17T17:59:24Z	INFO	Number of language-specific files	num=1
2024-10-17T17:59:24Z	INFO	[gobinary] Detecting vulnerabilities...
2024-10-17T17:59:24Z	DEBUG	[gobinary] Scanning packages for vulnerabilities	file_path="opt/bitnami/redis-exporter/bin/redis_exporter"
2024-10-17T17:59:24Z	DEBUG	[gobinary] Skipping vulnerability scan as no version is detected for the package	name="github.com/oliver006/redis_exporter"
2024-10-17T17:59:24Z	DEBUG	[vex] VEX filtering is disabled

bitnami/redis-exporter:1.43.0-debian-11-r4 (debian 11.3)

Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

The worst part here is that it fail silently, it doesn't say anywhere cannot scan or anything, it look like a perfectly legit scan, just found 0 CVEs

30min later, scanning the very same bitnami/redis-exporter:1.43.0-debian-11-r4 image will report 308 CVEs

 trivy image bitnami/redis-exporter:1.43.0-debian-11-r4 --scanners vuln --debug
2024-10-17T18:28:02Z	DEBUG	No plugins loaded
2024-10-17T18:28:02Z	DEBUG	Default config file "file_path=trivy.yaml" not found, using built in values
2024-10-17T18:28:02Z	DEBUG	Cache dir	dir="/home/app/.cache/trivy"
2024-10-17T18:28:02Z	DEBUG	Cache dir	dir="/home/app/.cache/trivy"
2024-10-17T18:28:02Z	DEBUG	Parsed severities	severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-10-17T18:28:02Z	DEBUG	Ignore statuses	statuses=[]
2024-10-17T18:28:02Z	DEBUG	DB update was skipped because the local DB is the latest
2024-10-17T18:28:02Z	DEBUG	DB info	schema=2 updated_at=2024-10-17T12:17:37.660183081Z next_update=2024-10-18T12:17:37.660182941Z downloaded_at=2024-10-17T18:26:35.719459154Z
2024-10-17T18:28:02Z	DEBUG	[pkg] Package types	types=[os library]
2024-10-17T18:28:02Z	DEBUG	[pkg] Package relationships	relationships=[unknown root direct indirect]
2024-10-17T18:28:02Z	INFO	[vuln] Vulnerability scanning is enabled
2024-10-17T18:28:02Z	DEBUG	Enabling misconfiguration scanners	scanners=[azure-arm cloudformation dockerfile helm kubernetes terraform terraformplan-json terraformplan-snapshot]
2024-10-17T18:28:02Z	DEBUG	Initializing scan cache...	type="fs"
2024-10-17T18:28:03Z	DEBUG	[image] Detected image ID	image_id="sha256:6a6d68284420f8ec7f3864464de8cf59510968954d22b1f0f4beb27ffa507966"
2024-10-17T18:28:03Z	DEBUG	[image] Detected diff ID	diff_ids=[sha256:9b35e8d8e8bf952b054748f36350bd3c6c3acd3c157d242b38019f90eb0a6a54 sha256:c667a923a11722e8e9fdd30675519973ec632dfbae7bf33d310004c0d552c215 sha256:9c8e78724ec633b64c0c0d5880ae00f4e838e3a343248799165c797e3daf9d04 sha256:1fe04252c67e8a5c9f403abc8639041f01e3a337b431ab9b55442b408e9905b4 sha256:00d41617810c06cbb85a472749e8da049c11a9877599a6518e929a33c4a0cb33 sha256:057558b45c7fea60a0a753389289dcd678892c0cb0dd1a538f5c593d12dd2ed0]
2024-10-17T18:28:03Z	DEBUG	[image] Detected base layers	diff_ids=[]
2024-10-17T18:28:03Z	INFO	Detected OS	family="debian" version="11.3"
2024-10-17T18:28:03Z	INFO	[debian] Detecting vulnerabilities...	os_version="11" pkg_num=110
2024-10-17T18:28:03Z	INFO	Number of language-specific files	num=1
2024-10-17T18:28:03Z	INFO	[gobinary] Detecting vulnerabilities...
2024-10-17T18:28:03Z	DEBUG	[gobinary] Scanning packages for vulnerabilities	file_path="opt/bitnami/redis-exporter/bin/redis_exporter"
2024-10-17T18:28:03Z	DEBUG	[gobinary] Skipping vulnerability scan as no version is detected for the package	name="github.com/oliver006/redis_exporter"
2024-10-17T18:28:03Z	WARN	Using severities from other vendors for some vulnerabilities. Read https://aquasecurity.github.io/trivy/v0.56/docs/scanner/vulnerability#severity-selection for details.
2024-10-17T18:28:03Z	DEBUG	[vex] VEX filtering is disabled

bitnami/redis-exporter:1.43.0-debian-11-r4 (debian 11.3)

Total: 308 (UNKNOWN: 0, LOW: 103, MEDIUM: 105, HIGH: 79, CRITICAL: 21)

And few hours later, it will again report 0 CVEs found

The only difference i see between those 2 scans, is that the one that found some issues had this new line

WARN	Using severities from other vendors for some vulnerabilities. Read https://aquasecurity.github.io/trivy/v0.56/docs/scanner/vulnerability#severity-selection for details.

Are there some quotas of vulnerabilities found from the same IP address ?? What could explain this discrepancy ?

Thanks a lot

Desired Behavior

Not having discrepancies

Actual Behavior

Mentioned in the description

Reproduction Steps

Mentioned in the description

Target

None

Scanner

None

Output Format

None

Mode

None

Debug Output

Mentioned in the description

Operating System

python:3.11.8-alpine3.19

Version

0.56.2

Checklist

• Run trivy clean --all
• Read the troubleshooting

[simar7]

That's interesting @krab-skunk - I've run Trivy in a loop for 30min or so but haven't seen this behavior yet. Where are you running Trivy?

[krab-skunk]

it run on a AWS, in a EKS cluster, base image is python:3.11.8-alpine3.19 . We are using trivy for almost 2 years now, but its just recently that we suddenly see some huge decrease in vulnerabilities count, so does our investigation that lead to find this issue.

Trivy may as well run correctly for a few hours, until it decide to suddenly flag every images with 0 CVE. And as said above, its dramatic for us, as there is no way for to check if its simply a real image with 0 CVEs or a scan error from trivy as it fails silently and the JSON payload does not display any errors, just no Vulnerabilities key in the Results Array.

[DmitriyLewen]

@krab-skunk Can you run Trivy with the -f json --list-all-pkgs flags and compare Packages when Trivy does and does not find vulnerabilities?

[krab-skunk]

it happened again today, we scan daily thousand of images , for example, we had a scan of grafana/agent:v0.43.3 and today it reported:

{
            "ArtifactName": "grafana/agent:v0.43.3",
            "ArtifactType": "container_image",
            "CreatedAt": "2024-11-14T01:03:30.756590876Z",
            "Metadata":
            {
                "DiffIDs":
                [
                    "sha256:ba8dbc5b24b59c2e6aff3639f32c5402af5e30be686342d08925ad852bd8c7c4",
                    "sha256:209530aab8f94b14edc36fc07c96fe8b82b1e65fade35554a6bf66b33f354519",
                    "sha256:bf49afc31bb9ae82f2dbf60e7a8bebe7534a9ae082757946f003fc6ee631dc78",
                    "sha256:8c9710d9bdf737a983b2105d482648c0488158912db615a8b44a734f18bf31e3",
                    "sha256:1cd48d8de194f7f94f90ae5eb4d456bd90857b61f14650e5cb78b7a8ec7bcbf9",
                    "sha256:59fd134921bfda4b433335e5c71dcc4f4d6fc62ec0150faa6de80f84f5e7d7da",
                    "sha256:d275ae91378e1c02dd5edba7558efc88185f14d4fd7bd681d3787a7738cecf56",
                    "sha256:7b40831cba0fd0b1788b8e250d986efa5c0d2f9c6e1f79b632118e0760d75887",
                    "sha256:7f495db269c5b253d175a95327bd4ee13a389c0a34ba17d8fbdd38001535693e"
                ],
                "ImageConfig":
                {
                    "architecture": "amd64",
                    "config":
                    {
                        "ArgsEscaped": true,
                        "Cmd":
                        [
                            "--config.file=/etc/agent/agent.yaml",
                            "--metrics.wal-directory=/etc/agent/data"
                        ],
                        "Entrypoint":
                        [
                            "/bin/grafana-agent"
                        ],
                        "Env":
                        [
                            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                            "AGENT_DEPLOY_MODE=docker"
                        ],
                        "Labels":
                        {
                            "org.opencontainers.image.ref.name": "ubuntu",
                            "org.opencontainers.image.source": "https://github.com/grafana/agent",
                            "org.opencontainers.image.version": "24.04"
                        }
                    },
                    "created": "2024-09-26T15:30:28.259718297Z",
                    "history":
                    [
                        {
                            "created": "2024-09-16T06:23:30.857138097Z",
                            "created_by": "/bin/sh -c #(nop)  ARG RELEASE",
                            "empty_layer": true
                        },
                        {
                            "created": "2024-09-16T06:23:30.887844049Z",
                            "created_by": "/bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH",
                            "empty_layer": true
                        },
                        {
                            "created": "2024-09-16T06:23:30.911509701Z",
                            "created_by": "/bin/sh -c #(nop)  LABEL org.opencontainers.image.ref.name=ubuntu",
                            "empty_layer": true
                        },
                        {
                            "created": "2024-09-16T06:23:30.934934561Z",
                            "created_by": "/bin/sh -c #(nop)  LABEL org.opencontainers.image.version=24.04",
                            "empty_layer": true
                        },
                        {
                            "created": "2024-09-16T06:23:32.858213294Z",
                            "created_by": "/bin/sh -c #(nop) ADD file:6f881131af38dde06e3705e8376701ae22ce479e4824821a9580d4e0e0bcf0ac in / "
                        },
                        {
                            "created": "2024-09-16T06:23:33.104112977Z",
                            "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
                            "empty_layer": true
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:24:43.996529673Z",
                            "created_by": "ARG UID=473",
                            "empty_layer": true
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:24:43.996529673Z",
                            "created_by": "ARG USERNAME=grafana-agent",
                            "empty_layer": true
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:24:43.996529673Z",
                            "created_by": "LABEL org.opencontainers.image.source=https://github.com/grafana/agent",
                            "empty_layer": true
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:24:43.996529673Z",
                            "created_by": "RUN |2 UID=473 USERNAME=grafana-agent /bin/sh -c   apt-get update\n  apt-get install -qy libsystemd-dev tzdata ca-certificates libcap2-bin\n  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*\n # buildkit"
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:27.435617388Z",
                            "created_by": "COPY /src/agent/build/grafana-agent /bin/grafana-agent # buildkit"
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:27.440433085Z",
                            "created_by": "COPY cmd/grafana-agent/agent-local-config.yaml /etc/agent/agent.yaml # buildkit"
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:27.595648614Z",
                            "created_by": "RUN |2 UID=473 USERNAME=grafana-agent /bin/sh -c groupadd --gid $UID $USERNAME # buildkit"
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:27.689118643Z",
                            "created_by": "RUN |2 UID=473 USERNAME=grafana-agent /bin/sh -c useradd -m -u $UID -g $UID $USERNAME # buildkit"
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:27.726301262Z",
                            "created_by": "RUN |2 UID=473 USERNAME=grafana-agent /bin/sh -c chown -R $USERNAME:$USERNAME /etc/agent # buildkit"
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:27.974961885Z",
                            "created_by": "RUN |2 UID=473 USERNAME=grafana-agent /bin/sh -c chown -R $USERNAME:$USERNAME /bin/grafana-agent # buildkit"
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:28.259718297Z",
                            "created_by": "RUN |2 UID=473 USERNAME=grafana-agent /bin/sh -c setcap 'cap_net_bind_service=+ep' /bin/grafana-agent # buildkit"
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:28.259718297Z",
                            "created_by": "ENTRYPOINT [\"/bin/grafana-agent\"]",
                            "empty_layer": true
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:28.259718297Z",
                            "created_by": "ENV AGENT_DEPLOY_MODE=docker",
                            "empty_layer": true
                        },
                        {
                            "comment": "buildkit.dockerfile.v0",
                            "created": "2024-09-26T15:30:28.259718297Z",
                            "created_by": "CMD [\"--config.file=/etc/agent/agent.yaml\" \"--metrics.wal-directory=/etc/agent/data\"]",
                            "empty_layer": true
                        }
                    ],
                    "os": "linux",
                    "rootfs":
                    {
                        "diff_ids":
                        [
                            "sha256:ba8dbc5b24b59c2e6aff3639f32c5402af5e30be686342d08925ad852bd8c7c4",
                            "sha256:209530aab8f94b14edc36fc07c96fe8b82b1e65fade35554a6bf66b33f354519",
                            "sha256:bf49afc31bb9ae82f2dbf60e7a8bebe7534a9ae082757946f003fc6ee631dc78",
                            "sha256:8c9710d9bdf737a983b2105d482648c0488158912db615a8b44a734f18bf31e3",
                            "sha256:1cd48d8de194f7f94f90ae5eb4d456bd90857b61f14650e5cb78b7a8ec7bcbf9",
                            "sha256:59fd134921bfda4b433335e5c71dcc4f4d6fc62ec0150faa6de80f84f5e7d7da",
                            "sha256:d275ae91378e1c02dd5edba7558efc88185f14d4fd7bd681d3787a7738cecf56",
                            "sha256:7b40831cba0fd0b1788b8e250d986efa5c0d2f9c6e1f79b632118e0760d75887",
                            "sha256:7f495db269c5b253d175a95327bd4ee13a389c0a34ba17d8fbdd38001535693e"
                        ],
                        "type": "layers"
                    }
                },
                "ImageID": "sha256:6688ac33665a062fb1a3d23afa727f65d46f294dc125241d1bf8c0a9e684b4ed",
                "OS":
                {
                    "Family": "ubuntu",
                    "Name": "24.04"
                },
                "RepoDigests":
                [
                    "grafana/agent@sha256:7f362921b4aabf1e8232ba0d33aba982b51793aa05675c8bee0511dbf77167e3"
                ],
                "RepoTags":
                [
                    "grafana/agent:v0.43.3"
                ]
            },
            "Results":
            [
                {
                    "Class": "os-pkgs",
                    "Target": "grafana/agent:v0.43.3 (ubuntu 24.04)",
                    "Type": "ubuntu"
                },
                {
                    "Class": "lang-pkgs",
                    "Target": "usr/bin/grafana-agent",
                    "Type": "gobinary"
                }
            ],
            "SchemaVersion": 2
        }

Results Key is empty , we re-run it again, and suddenly all CVE re-appeared, its happen so randomly, I'll try to see if i can enable debug --list-all-pkgs , but its so random, its super hard to catch
actually , today more than 100 images return an empty scan while it shouldn't :( when a scan fail, we should never return what appears to be a valid payload

[DmitriyLewen]

I'll try to see if i can enable debug --list-all-pkgs

I tried to find a place where the problem might be, but I didn't find it.
I asked about --list-all-pkgs flag - because it will narrow down the search area (for example, Trivy incorrectly detects package versions or doesn't find vulnerabilities).

---

one more question: do you run multiple Trivy binaries in parallel or do you always have just one Trivy instance running?