"{"errorMessage":"Message field name is too long"}

[dpallagolla]

Describe the bug

Few logs are skipped from loginsight instance with the error message "{"errorMessage":"Message field name is too long"}, there is no retry attempted from the plugin for this.

Reproduction steps

1. Install the fluentd plugin on any kubernetes node
   2.Enabled https debug logs
   3.Observe that few calls to vRLI fail with "{"errorMessage":"Message field name is too long"}
   ...

Expected behavior

logs should be forwarded to vRLI without any errors

Additional context

No response

[dpallagolla]

Found the root cause, if the labels being sent from fluentd to loginsight is longer than 64 characters, the payload is rejected by loginsight.

Because of this, logs from other pods may go missing as one post call to loginsight would involve events from multiple pods

Below is a sample payload that will be rejected by loginsight.

{
  "events": [
    {
      "fields": [
        {
          "name": "k8s_pod",
          "content": "calico-node-c4hd7"
        },
        {
          "name": "k8s_annotations_kubeadm_k8s_io_kube_apiserver_advertise_address_endpoint",
          "content": "kube-system"
        },
       
      ],
      "text": "text",
      "timestamp": 1714024595000
    }
  ]
}

Although there is the shorten_keys configuration available, it would be nice to have some feature that would truncate labels based on length criteria. It may not always be feasible to use shorten_keys config.

[Griamce0385]

We ran into this issue as well and would apricate support for names longer then 64 Char.