Scan Kubernetes Custom Resources?

[tschran]

Question

Is there a way to configure the Kubernetes scanner to understand custom resources? We're trying to scan some clusters that are running Argo Rollouts and Argo Workflows. Both of those services use Custom Resources that define resources to create. Rollouts is basically a replacement for Deployments, so it will create and manage ReplicaSets, and Argo Workflows creates a bunch of pods. Right now, I'm getting separate reports for hundreds of Argo Workflow related pods, even though they're all coming from the same source. This, in turn, is causing the scan run to get OOMKilled frequently, because the run is producing a report that's over a gigabyte.
For the built-in resources, it looks like you're rolling up and deduping the scan findings. For instance, there may be a hundred pods on a cluster, but they all come from the same Deployment, so Trivy just reports one set of findings against that Deployment. I'd like to do the same thing for these custom resources.

Target

Kubernetes

Scanner

Vulnerability

Output Format

JSON

Mode

Standalone

Operating System

Ubuntu 22.04

Version

Version: 0.56.2
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-10-25 12:17:51.722810731 +0000 UTC
  NextUpdate: 2024-10-26 12:17:51.72281046 +0000 UTC
  DownloadedAt: 2024-10-25 15:58:00.246567921 +0000 UTC
Java DB:
  Version: 1
  UpdatedAt: 2024-10-23 02:37:26.813900849 +0000 UTC
  NextUpdate: 2024-10-26 02:37:26.813900579 +0000 UTC
  DownloadedAt: 2024-10-23 19:51:06.355657256 +0000 UTC
Check Bundle:
  Digest: sha256:ef2d9ad4fce0f933b20a662004d7e55bf200987c180e7f2cd531af631f408bb3
  DownloadedAt: 2024-09-18 20:21:26.698079409 +0000 UTC

[simar7]

AFAIK With trivy k8s scanning, you are only able to scan core Kubernetes components, so scanning custom resources might not be possible, correct me if I'm wrong @afdesk

Although having said that, if you're looking to scan custom resources with the trivy config misconfiguration scanner (i.e. scanning deployment YAML files, for instance) you could write your own schema and checks to be able to support custom resources like described here https://aquasecurity.github.io/trivy/v0.56/docs/scanner/misconfiguration/custom/schema/#custom-checks-with-custom-schemas

[afdesk]

@simar7 you're right. Trivy k8s doesn't scan custom resources now.
Trivy receives only next resources from k8s: https://github.com/aquasecurity/trivy-kubernetes/blob/b6099eb824b5be348f257560f2544ce20eeb6031/pkg/k8s/k8s.go#L406-L432

[itaysk]

it should be possible to create a custom check (for Argo resources) also without schema, isn't it?
there's a demo of this here: https://youtu.be/nGcvPAQdpVg?si=qXL9Tjkxfx_BRMak&t=545 some things might have changed since, but in principle it should work

[simar7]

@itaysk the demo uses a custom schema like I mentioned.

The project structure uses Rego convention so Trivy is automatically able to pick up the schema. This is documented here

[tschran]

We're only using the vulnerability scanner at this point.