All pods are reported as "deleted": true even if still running

[headyj]

Describe the bug
On one of our cluster, the JSON export reports all pods as "deleted": true even if they are still running:

To Reproduce
Steps to reproduce the behavior:

1. Execute the following command:

krr simple --logtostderr \
 --namespace kube-system \
 -f json --history_duration 720 \
 --fileoutput /tmp/krr-output.json

1. Check the output, in some cases all the pods are reported as deleted:

{
  "scans": [
  [...]
    {
      "object": {
        "cluster": "my-cluster",
        "name": "ebs-csi-controller",
        "container": "ebs-plugin",
        "pods": [
          {
            "name": "ebs-csi-controller-7899c58f65-2cjh9",
            "deleted": true
          },
          {
            "name": "ebs-csi-controller-7899c58f65-jr7h6",
            "deleted": true
          },
          {
            "name": "ebs-csi-controller-65599649b8-sr67x",
            "deleted": true
          },
          {
            "name": "ebs-csi-controller-7899c58f65-fsbc5",
            "deleted": true
          },
          {
            "name": "ebs-csi-controller-7899c58f65-zfpqg",
            "deleted": true
          },
          {
            "name": "ebs-csi-controller-7899c58f65-rd4hh",
            "deleted": true
          },
          {
            "name": "ebs-csi-controller-65599649b8-g9cjk",
            "deleted": true
          }
        ],
[...]

Even if some of them are still running:

kubectl get pods -n kube-system | grep ebs-csi-controller
ebs-csi-controller-7899c58f65-2cjh9          6/6     Running   0          43h
ebs-csi-controller-7899c58f65-rd4hh          6/6     Running   0          43h

For any reason, we have this issue only one cluster out of 3. All of them are on running the same version (v1.29) and are deployed using EKS and the same version of Prometheus. I also tried on multiple namespaces and on this specific namespace, all the pods are reported as deleted.

Expected behavior
deleted should be set to false if the pod is still running

Are you interested in contributing a fix for this?
Not really :(

Desktop (please complete the following information):

• OS: Ubuntu
• Browser: N/A
• Version: N/A

Additional context

• krr v1.11.0
• Kubernetes v1.29 (EKS)

[headyj]

As usual, I found the problem one minute after reporting the issue.... For those who are having the same issue, check that kube_pod_status_phase is returning data on prometheus. For any reason one of our prometheus was not returning data by default...