From a0df6d44f45b831865943a78f55a9a8b34f07b84 Mon Sep 17 00:00:00 2001 From: Chris Foster Date: Thu, 22 Jun 2017 17:45:29 -0700 Subject: [PATCH] Start describing proposed implementation * Quickstart for frontend * What is a log record? * Some work on inspiration section --- Logging.md | 122 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 106 insertions(+), 16 deletions(-) diff --git a/Logging.md b/Logging.md index 6ad806a..a21e9bd 100644 --- a/Logging.md +++ b/Logging.md @@ -47,14 +47,15 @@ rather than `println()`. We should have: stack trace, etc.) should be automatically gathered without a syntax burden. * Freedom in formatting the log message - simple string interpolation, `@sprintf` and `fmt()`, etc should all be fine. -* Zero logger setup for simple uses. +* No mention of log dispatch should be necessary at the message creation site. * Easy filtering of log messages. * Clear guidelines about the meaning and appropriate use of standard log levels for consistency between packages, along with guiding the appropriate use of logging vs stdout. -* To encourage the use of logging over ad hoc console output, the default - console log handler should emphasize the *log message*, and metadata should be - communicated in a non-intrusive way. +* The default console log handler should integrate somehow with the display + system to show log records in a way which is highly readable. Ideally logging + should be a tool for package authors just as much as it is a tool for + understanding production systems. The API should be **flexible enough** for advanced users: @@ -94,8 +95,82 @@ comes in three flavours: ## Proposed design -A rough work in progress implementation is available at -https://github.com/c42f/MicroLogging.jl +A prototype implementation is available at https://github.com/c42f/MicroLogging.jl +### Quickstart Example + +#### Frontend +```julia +# using Base.Log + +# Logging macros +@debug "A message for debugging (filtered out by default)" +@info "Information about normal program operation" +@warn "A potentially problem was detected" +@error "Something definitely went wrong, but we recovered enough to continue" +@logmsg Log.Info "Explicitly defined info log level" + +# Free form message formatting +x = 10.50 +@info "$x" +@info @sprintf("%.3f", x) +@info begin + A = ones(4,4) + "sum(A) = $(sum(A))" +end + +# Progress reporting +for i=1:10 + @info "Some algorithm" progress=i/10 +end + +# User defined key value pairs +foo_val = 10.0 +@info "test" foo=foo_val bar=42 +``` + + +### What is a log record? + +Logging statements are used to understand algorithm flow - the order and timing +in which logging events happen - and the program state at each event. Each +logging event is preserved in a **log record**. The information in a record +needs to be gathered efficiently, but should be rich enough to give insight into +program execution. + +A log record includes information explicitly given at the call site, and any +relevant metadata which can be harvested from the lexical and dynamic +environment. Most logging libraries allow for two key pieces of information +to be supplied explicitly: + +* The **log message** - a user-defined string containing key pieces of program + state, chosen by the developer. +* The **log level** - a category for the message, usually ordered from verbose + to severe. The log level is generally used as an initial filter to remove + verbose messages. + +Some logging libraries (for example +[glib](https://developer.gnome.org/glib/stable/glib-Message-Logging.html) +structured logging) allow users to supply extra log record information in the +form of key value pairs. Others like +[log4j2](https://logging.apache.org/log4j/2.x/manual/messages.html) require extra information to be +explicitly wrapped in a log record type. In julia, supporting key value pairs +in logging statements gives a good mixture of usability and flexibility: +Information can be communicated to the logging backend as simple keyword +function arguments, and the keywords provide syntactic hints for early filtering +in the logging macro frontend. + +In addition to the explicitly provided information, some useful metadata can be +automatically extracted and stored with each log record. Some of this is +extracted from the lexical environment or generated by the logging frontend +macro, including code location (module, file, line number) and a unique message +identifier. The rest is dynamic state which can generally on demand by the +backend, including system time, stack trace, current task id. + +### The logging frontend + +### Dispatching records + +### Early filtering ## Concrete use cases @@ -113,19 +188,34 @@ both in the code and for user control. * The depwarn system, and `--depwarn` command line flag -## Survey of other logging systems +## Inspriation -Much of the above is a reinterpretation in julia of python's standard logging -module, attempting to be even more convenient and much more efficient, while -retaining the same flexibility. +This Julep draws inspiration from many previous logging frameworks, and helpful +discussions with many people online and at JuliaCon 2017. -* A summary of how this all fits in with Logging.jl, MiniLogging.jl, Memento.jl, - LumberJack.jl, and any other logging libraries which can be found. -* glib (C) - https://developer.gnome.org/glib/stable/glib-Message-Logging.html +The Java logging framework [log4j2](https://logging.apache.org/log4j/2.x/) was a +great source of use cases, as it contains the lessons from at least twenty years +of large production systems. While containing a fairly large amount of +complexity, the design is generally very well motivated in the documentation, +giving a rich set of use cases. The julia logging libraries - Base in julia 0.6, +Logging.jl, MiniLogging.jl, LumberJack.jl, and particularly +[Memento.jl](https://github.com/invenia/Memento.jl) - provided helpful +context for the needs of the julia community. + +Structured logging as available in +[glib](https://developer.gnome.org/glib/stable/glib-Message-Logging.html) +and [RFC5424](https://datatracker.ietf.org/doc/rfc5424/?include_text=1) (The +Syslog protocol) provide context for the usefulness of log records as key value +pairs. + +For the most part, existing julia libraries seem to follow the design tradition +of the standard [python logging library](https://docs.python.org/3/library/logging.html), +which has a lineage further described in [PEP-282](https://www.python.org/dev/peps/pep-0282/). +The python logging system provided a starting point for this Julep, though the +design eventually diverged from the typical hierarchical setup. + +TODO: Re-survey the following? * a-cl-logger (Common lisp) - https://github.com/AccelerationNet/a-cl-logger * Lager (Erlang) - https://github.com/erlang-lager/lager -* Some java logger *cough*? That should be enterprisey enough, right ;-) -* syslog (unix) ? -TODO - much more here.