Reckless is an extremely low-latency, high-throughput logging library. It was created because I needed to perform extensive diagnostic logging without worrying about performance. Other logging libraries boast the ability to throw log messages away very quickly. Reckless boasts the ability to keep them all, without worrying about the performance impact. Filtering can and should wait until you want to read the log, or need to clean up disk space.
By low latency I mean that the time from calling the library and returning to the caller is as short as I could make it. The code generated at the call site consists of
- Pushing the arguments on a thread-local queue. This has the same cost as pushing the arguments on the stack for a normal function call.
- Call to Boost.Lockless (NB: this is bundled with the library, not an external dependency) to register the write request on a shared lockless queue.
The actual writing is performed asynchronously by a separate thread. This removes or hides several costs:
- No transition to the kernel at the call site. The kernel is an easily overlooked but important cost, not only because the transition costs time, but because it pollutes the CPU cache. In other words, avoiding this makes your non-logging code run faster than if you were using a library that has to enter the kernel to perform its work.
- No locks need to be taken for synchronization between threads (unless the queue fills up; see the performance article for more information about the implications of this).
- It doesn't have to wait for the actual I/O operation to complete.
- If there are bursts of log calls, multiple items on the queue can be batched into a single write, improving throughput without sacrificing write latency.
For a more detailed performance discussion and statistics, see the performance article.
As all string formatting and I/O is done asynchronously and in a single thread, there are a few caveats you need to be aware of:
- If you choose to pass log arguments by reference or pointer, then you
must ensure that the referenced data remains valid at least until the
log has been flushed or closed (unless you're only interested in
logging the value of the pointer itself). The best option for
dynamically allocated data is typically
std::string
orstd::shared_ptr
. - You must take special care to handle crashes if you want to make sure that all log data prior to the crash is saved. This is not unique to asynchronous logging—for example fprintf will buffer data until you flush it—but asynchronous logging arguably makes the issue worse. The library provides convenience functions to aid with this.
- As all string formatting is done in a single thread, it could theoretically limit the scalability of your application if the formatting is very expensive.
- Performance becomes somewhat less predictable. Rather than putting the cost of the logging on the thread that calls the logging library, the OS may suspend some other thread to make room for the logging thread to run.
#include <reckless/severity_log.hpp>
#include <reckless/file_writer.hpp>
// It is possible to build custom loggers for various ways of formatting the
// log. The severity log is a stock policy-based logger that allows you to
// configure fields that should be put on each line, including a severity
// marker for debug/info/warning/error.
using log_t = reckless::severity_log<
reckless::indent<4>, // 4 spaces of indent
' ', // Field separator
reckless::severity_field, // Show severity marker (D/I/W/E) first
reckless::timestamp_field // Then timestamp field
>;
reckless::file_writer writer("log.txt");
log_t g_log(&writer);
int main()
{
std::string s("Hello World!");
g_log.debug("Pointer: %p", s.c_str());
g_log.info("Info line: %s", s);
for(int i=0; i!=4; ++i) {
reckless::scoped_indent indent; // The indent object causes the lines
g_log.warn("Warning: %d", i); // within this scope to be indented
}
g_log.error("Error: %f", 3.14);
return 0;
}
This would give the following output:
D 2015-03-29 13:23:35.288 Pointer: 0x1e18218
I 2015-03-29 13:23:35.288 Info line: Hello World!
W 2015-03-29 13:23:35.288 Warning: 0
W 2015-03-29 13:23:35.288 Warning: 1
W 2015-03-29 13:23:35.288 Warning: 2
W 2015-03-29 13:23:35.288 Warning: 3
E 2015-03-29 13:23:35.288 Error: 3.140000
The library currently works only on Linux. Windows and BSD are on the roadmap. I don't own any Apple computers, so OS X won't happen unless someone sends me a patch or buys me hardware.
To build the library, clone the git repository and run make.
To build a program against the library, given the variable RECKLESS pointing to the reckless root directory, use:
g++ -std=c++11 myprogram.cpp -I$(RECKLESS)/reckless/include -L$(RECKLESS)/reckless/lib -lreckless
For more details, see the manual.
Two other logging libraries with a similar, asynchronous design are spdlog and g2log.