DB logger sample

[matejk]

Resolves #4750

[aleks-f]

Functionality ok, but it should be tidied up a bit, there's too much functionality crammed into a single class. Something along these lines:

poco/Util/samples/SampleServer/src/SampleServer.cpp

Lines 108 to 112 in c8dade8

	TaskManager tm;
	tm.start(new SampleTask);
	waitForTerminationRequest();
	tm.cancelAll();
	tm.joinAll();

[matejk]

Functionality ok, but it should be tidied up a bit, there's too much functionality crammed into a single class. Something along these lines:

poco/Util/samples/SampleServer/src/SampleServer.cpp

Lines 108 to 112 in c8dade8

	TaskManager tm;
	tm.start(new SampleTask);
	waitForTerminationRequest();
	tm.cancelAll();
	tm.joinAll();

Extracted scanning and inserting functionality to a separate class SQLLogInserter.

Is that what you had in mind by tidying up?

[aleks-f]

Need to have DBLogger.properties file, with all the commend option settings and logger settings, something like this:

logging.loggers.root.channel = appSplitter
logging.loggers.root.level = information

logging.channels.appFile.class = FileChannel
logging.channels.appFile.pattern = %L%Y-%m-%d %H:%M:%S.%i [%p] %s<%I>: %t
logging.channels.appFile.path = ${application.baseName}.log
logging.channels.appFile.times = local
logging.channels.appFile.rotation = daily
logging.channels.appFile.archive = number
logging.channels.appFile.purgeCount = 10

logging.channels.sql.class = SQLChannel
logging.channels.sql.name = DBLogger
logging.channels.sql.connector = SQLite
logging.channels.sql.connect = DBLogger.db3
logging.channels.sql.timeout = 3000
logging.channels.sql.minBatch = 5
logging.channels.sql.maxBatch = 1000
logging.channels.sql.bulk = true
logging.channels.sql.table = T_LOG
logging.channels.sql.async = false
logging.channels.sql.throw = false
logging.channels.sql.file = ${application.baseName}.sql

logging.channels.console.class = ColorConsoleChannel
logging.channels.console.pattern = %L%Y-%m-%d %H:%M:%S.%i [%p] %s<%I>: %t

logging.channels.appSplitter.class = SplitterChannel
logging.channels.appSplitter.channels = console, appFile, sql

[aleks-f]

• Sample should work out of the box, but it doesn't:

$ bin/Linux/aarch64/DBLogger
Directory: 
Database: , 
Number of workers: 2
Not found
Created 0 messages, processed 0 messages in 0 ms.

• There should be a reasonable initial default setup so that workers can keep up with sql file generation; as it is now, the filesystem gets clogged with sql files.

• logging configuration, as mentioned here

• SQL statements in the files should insert more than a single row; should be achievable with minBatch property

[matejk]

* [ ]  SQL statements in the files should insert more than a single row; should be achievable with `minBatch` property

@aleks-f :

Is it a matter of SQL channel configuration to insert multiple SQL statements into the same file or shall DB logger have its own configuration to merge multiple sql files together?

[aleks-f]

* [ ]  SQL statements in the files should insert more than a single row; should be achievable with `minBatch` property

@aleks-f :

Is it a matter of SQL channel configuration to insert multiple SQL statements into the same file or shall DB logger have its own configuration to merge multiple sql files together?

Number of rows inserted in one statement should be controllable with minBatch, maxBatch and maxSQL property, when >1 the channel should concatenate multiple rows so they are inserted in one execution:

poco/Data/src/SQLChannel.cpp

Lines 56 to 58 in cb91880

	const std::string SQLChannel::SQL_INSERT_STMT = "INSERT INTO %s " \
	"(Source, Name, ProcessId, Thread, ThreadId, Priority, Text, DateTime)" \
	" VALUES %s";

The maxSQL property should be set not to exceed the max SQL statement length allowed by back end:

poco/Data/src/SQLChannel.cpp

Lines 400 to 408 in cb91880

	if (++batch == _maxBatch || (os.str().length() + tmp.str().length()) >= _maxSQL)
	{
	os << ";\n";
	doLog();
	os.str(""); sql.clear();
	Poco::format(sql, SQL_INSERT_STMT, _table, std::string());
	os << sql << '\n' << tmp.str();
	batch = 0;
	}

See eg. Batch size for SQL Server

[matejk]

* [ ]  SQL statements in the files should insert more than a single row; should be achievable with `minBatch` property

@aleks-f :
Is it a matter of SQL channel configuration to insert multiple SQL statements into the same file or shall DB logger have its own configuration to merge multiple sql files together?

Number of rows inserted in one statement should be controllable with minBatch, maxBatch and maxSQL property, when >1 the channel should concatenate multiple rows so they are inserted in one execution:

Thanks, the latest code uses SQLLogger demo channel configuration from DBLogger.properties where all of its properties can be tuned.