Using sc4s alternate destination config to forward syslog to a thirdparty SIEM do not retain the source IP after being being received by a SIEM

[aalisher-tmx]

Was the issue replicated by support?
Yes. Case #3475996
What is the sc4s version ?
3.16.0
Which operating system (including its version) are you using for hosting SC4S?
Rhel 8
Which runtime (Docker, Podman, Docker Swarm, BYOE, MicroK8s) are you using for SC4S?
podman
Is there a pcap available? If so, would you prefer to attach it to this issue or send it to Splunk support?
Yes, will send seperately once the issue is assigned
Is the issue related to the environment of the customer or Software related issue?
Software related
Is it related to Data loss, please explain ?
Protocol? Hardware specs?

Last chance index/Fallback index?

Is the issue related to local customization?

Do we have all the default indexes created?

Describe the bug
A clear and concise description of what the bug is.
All our syslog devices are ingesting syslog to Splunk Cloud via sc4s app using HEC. No issues. All good. Also, using alternate destinations config (https://splunk.github.io/splunk-connect-for-syslog/main/destinations/#example-5-cisco-asa-send-to-a-third-party-siem), we want to forward to the third-party SIEM (QRADAR) appliance. I followed the documentation and logs are showing in qradar, but with the following issue:

• The source IP is being overwritten with the sc4s server instead of the actual source IP after being received by a SIEM from sc4s. - client IP ----> sc4s (syslog-ng) ---> SIEM (source is coming up as sc4s server instead of client IP).
• looks like the syslog protocol is unable to maintain the original sender's address when forwarding syslog messages. This is because the sender's address is taken from the received TCP packet.

env_file:

SC4S_DEST_SYSLOG_QRADARSIEM_HOST=x.x.x.x
SC4S_DEST_SYSLOG_QRADARSIEM_PORT=514
SC4S_DEST_SYSLOG_QRADARSIEM_MODE=SELECT

set to #yes for ietf frames

SC4S_DEST_SYSLOG_QRADARSIEM_IETF=yes

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

[aalisher-tmx]

Hi @rjha-splunk any update on this? Thanks

[rjha-splunk]

Hi @aalisher-tmx unfortunately syslog-ng under the hood retains the last relays ip as IP of the source, in this case it is SC4S https://syslog-ng.github.io/admin-guide/110_Template_and_rewrite/000_Customize_message_format/004_Macros_of_syslog-ng#sourceip

I can still send this information using templates and reset the soft MACRO host with source ip here

Example env_config:
SC4S_DEST_SYSLOG_TEST_HOST=x.x.x.x
SC4S_DEST_SYSLOG_TEST_PORT=514
SC4S_DEST_SYSLOG_TEST_MODE=SELECT
SC4S_DEST_SYSLOG_TEST_TRANSPORT=UDP
SC4S_SOURCE_STORE_RAWMSG=yes

Example conf file:

block parser sc4s-lp-cisco_asa_d_syslog_test(){
    channel {
        rewrite {
        set('$SOURCEIP', value('HOST'));
    };
 };

};


application sc4s-lp-cisco_asa_d_syslog_test[sc4s-lp-dest-select-d_syslog_test] {
    filter {
        'cisco' eq "${fields.sc4s_vendor}" and 'asa' eq "${fields.sc4s_product}";
    };

    parser {
        sc4s-lp-cisco_asa_d_syslog_test();
        syslog-parser(template("$MESSAGE"));
    };
};

[aalisher-tmx]

No, it didn't work @rjha-splunk
SIEM is looking into the header info and calculating the source ip as 10.152.210.206 (which is the sc4s server instead of the actual host sending the packet).

As per your config, host is being changed to IP in the sample message content shown below (see 10.152.208.98), but we don't want that

Sample Packet Header:
18:21:57.494911 IP (tos 0x0, ttl 64, id 61068, offset 0, flags [DF], proto TCP (6), length 159)
10.152.210.206.41025 > 10.152.1.129.514: Flags [P.], cksum 0xe9fc (incorrect -> 0x470d), seq 2382849015:2382849122, ack 1794184352, win 58, options [nop,nop,TS val 1786126440 ecr 2148056678], length 107

Sample Message:
1.2024-06-24T22:21:57.494+00:00.10.152.208.98.-.6736.-.-....0].info:.start.of.service.(unbound.1.6.6)..

[rjha-splunk]

We need to look for a solution and possible enhancement, we will keep you posted

[rjha-splunk]

I investigated and checked, right now i have following for the case:

1. we cant make it possible for TCP, UDP we can try to implement that using spoof-source and python parsers but it will be only possible after syslog-ng 4.8.0 release

2. Right now using parsers Sourceip information can be added to the header.

application sc4s-lp-cisco_asa_d_syslog_test[sc4s-lp-dest-select-d_syslog_test] {
    filter {
        'cisco' eq "${fields.sc4s_vendor}" and 'asa' eq "${fields.sc4s_product}";
    };

    parser {
        syslog-parser(template("SourceIP=${SOURCEIP} ${MSG}\n"));
    };
};

[rjha-splunk]

Closing this issue as per above comment, SC4S today cant spoof tcp packets intial origin source.

[aalisher-tmx]

2. Right now using parsers Sourceip information can be added to the header.

application sc4s-lp-cisco_asa_d_syslog_test[sc4s-lp-dest-select-d_syslog_test] {
    filter {
        'cisco' eq "${fields.sc4s_vendor}" and 'asa' eq "${fields.sc4s_product}";
    };

    parser {
        syslog-parser(template("SourceIP=${SOURCEIP} ${MSG}\n"));
    };
};

was this tested? above config doesn't update the header source ip.

[rjha-splunk]

Yes it is tested with ASA sample, it will just add sourceip to the message , i will share the screenshot tomorrow

[aalisher-tmx]

Oh, adding sourceip to the message not to the header. This doesn't resolve our issue.
Can you please let us know if there are any timelines to get my issue resolved? Thanks

[aalisher-tmx]

documentation also needs to be updated appropriately as this is not working as expected - https://splunk.github.io/splunk-connect-for-syslog/main/destinations/#example-5-cisco-asa-send-to-a-third-party-siem

[rjha-splunk]

As mentioned in my previous comment, right now syslog-ng under the hood doesn't provide this capability, we are working with them to get a work around.

For now alternate destinations can be explored using edge processor( splunk)

SC4S can be used to send data to third party but it will be written as source because Syslog does not include this information. We do embed it into otlp as shown but the source is lost as soon as it hops

[aalisher-tmx]

Thanks @rjha-splunk

[aalisher-tmx]

Just to confirm, splunk edge processor can be used to ingest UF/hec/sc4s based logs to splunkcloud or splunk enterprise indexes or AWS S3, but cannot ingest to any alternate destination (for e.g SIEM). Its NOT supported

https://docs.splunk.com/Documentation/SplunkCloud/9.2.2403/EdgeProcessor/ManageDestination