f5:bigip:syslog sourcetype falls back to nix:syslog when line starts with -c

[PashFW]

due to the known bug of f5 ...
Bug ID 1057305: On deployments that use DPDK, "-c" may be logged as the TMM process/thread name.
cdn.f5.com/product/bugtracker/ID1057305.html

...some syslog lines may start with "-c[123]" instead of the expected common "tmm[123]", which causes sc4s to mark these structurally valid lines as nix:syslog, causing incorrect further parsing in splunk

probably it makes sense to add a workaround by pattern like ^-c\[\d+\] to avoid transforms on the splunk end

example of normal line
tmm1[10801]: Rule /Common/http_log <HTTP_RESPONSE>: 22:20:48 | Client IP: 137.86.211.3 | HTTP Request Method: GET | HOSTNAME: mbcu-rm.example.com | URL: /api/order/creative_actions | Response Code: 200 | Cookie:

example of mismatched line
-c[10801]: Rule /Common/ops207709_api_irule <HTTP_RESPONSE>: 22:20:48 | Client IP: 34.219.299.167 | HTTP Request Method: GET | HOSTNAME: api.example.com | URL: /services/v4/programmatic/deals/110209 | Response Code: 200 | Cookie:

[cwadhwani-splunk]

Hi @PashFW ,

We’ve looked into the issue. The reason that the f5:bigip:syslog sourcetype falls back to nix:syslog when the line starts with "-c" is that SC4S parser is designed to recognize the program value prefix as 'TMM'. However, because of the "-c" at the start, it's not able to identify it correctly, and it ends up categorizing the event under nix:syslog.

We can create a workaround by providing a local parser to fix this. Could you clarify your preference? Do you want the sourcetype to stay as f5:bigip:syslog for events that start with "-c", or would you prefer us to drop those events before forwarding them to Splunk? We can set up a postfilter parser based on your needs.

[PashFW]

I'm fine, doing sourcetype rewrites later down the pipeline.
It rather makes sense to fix permanently for the common good.
These -c events are valid i.e. shall not be dropped, but properly sourcetype-d.

[cwadhwani-splunk]

You can use the following postfilter in your local environment to handle events that start with -c.

Steps to Update

1. Navigate to the directory where SC4S custom configurations are stored:cd /opt/sc4s/local/config/app_parsers
2. Create a new file named app-postfilter-f5_bigip_syslog.conf and paste the provided code into it.

block parser app-postfilter-f5_bigip_syslog() {
    channel {
        rewrite {
            r_set_splunk_dest_default(
                index("netops")
                source("program:${PROGRAM}")
                sourcetype('f5:bigip:syslog')
                vendor('f5')
                product('bigip')
            );
        };
    };
};

application app-postfilter-f5_bigip_syslog[sc4s-postfilter] {
    filter {
        program('-c' type(string) flags(prefix))
    };
    parser { app-postfilter-f5_bigip_syslog(); };
};

3. Restart the SC4S service to apply the changes: sudo systemctl restart sc4s

This issue originates from the Big IP vendor. Our team will discuss and evaluate incorporating this fix into the SC4S codebase as a permanent solution. Let us know if you have any further questions!