error_class=JSON::GeneratorError error="source sequence is illegal/malformed utf-8

[rahmisacal]

Describe the bug

when i try to send some infrastructure logs from openshift container platform, i run into the error like "error_class=JSON::GeneratorError error="source sequence is illegal/malformed utf-8".

I added char_encoding utf-8 and char_encoding binary:utf-8 as record_modifier however I could not get rid of this issue. you can see the Fluentd configuration as below:

--
data:
fluent.conf: |-

@type forward
port 20001
bind 0.0.0.0

<filter *.**>
  @type record_modifier
  char_encoding utf-8
  <record>
    environment infra
  </record>
</filter>

<match **>
   @type vmware_loginsight
   scheme https
   ssl_verify false
   host ****
   port ****
   http_method post
   serializer json
   raise_on_error true
   http_conn_debug false
   <buffer>
     @type file
     path /var/log/fluent/
     timekey 60
     flush_thread_count 24
     queued_chunks_limit_size 2048
     chunk_limit_size 16MB
     retry_type periodic
     retry_forever true
     retry_wait 5
     timekey_wait 0
   </buffer>
</match>

In addition I have no any problem when sending application logs and audit logs for now, only when sending infrastructure logs.

Reproduction steps

1. forward infrastructure logs to fluentd
2. send infrastructure logs with Fluentd to VMware loginsight
3. see this error message
   ...

Expected behavior

roughly a month ago, i have no any case when sending infrastructure logs with the same configuration but now I can not send infrastructure logs

Additional context

No response