Not able to concatenate multiline log separated in multiple events.

[MridulaSivanandan]

Problem

• An xml event is tailed by fluentbit and sent as multiple events to fluentd.
• Trying to concate them to single event using the concat plugin but it remains as multiple events
• The requirement is to get the whole xml data between an open and close tag as single event.
• The tag is <testsuite ..> {data} </testsuite>

Steps to replicate

Please try out the config below for the sample data given.

Concat Plugin Config

<filter testresults.**>
  @type concat
  key log
  multiline_start_regexp /^<testsuite/
  max_lines 4000
  log_level debug
  timeout_label @TEST
</filter>

Sample Data:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="Test" tests="6" skipped="0" failures="0" errors="0" timestamp="2019-08-14T09:42:30" hostname="users" time="1.054">
 <properties/>
 <testcase name="missingLatitude()" classname="ControllerTest" time="0.438"/>
 <testcase name="incorrectly()" 
classname="ControllerTest" time="0.125"/>
 <system-out><<![CDATA[2019-08-14 15:12:23.812  INFO 30116]]></system-out>
 <system-err><![CDATA[]]></system-err>
</testsuite>

Expected Behavior

• The whole data between <testsuite *> </testsuite> should be output as single event from fluentd
  Please comment if there are better ways of achieving this result

Your environment

• OS version - Ubuntu 18.10
• paste result of fluentd --version or td-agent --version - td-agent 1.4.2
• plugin version >= 2.0.0

[okkez]

Try using multiline_end_regexp.

[kaisermario]

@okkez why using multiline_end_regexp. Does multiline_start_regexp not work? There seemed to be many open issues regarding concatenation of multineline logs. Can somebody explain the current status of this project?

[okkez]

See test cases around multi_line_start_regexp and multi_line_end_regexp.