Copyright (C) 2022, TomTom International BV

This class library contains a number of useful tools when coding in Kotlin. The code has been made open-source by TomTom so others can use it and contribute to it.

Currently, the library contains:

• traceevents - This is a library to log semantic events in an application, where perhaps normally logging and log analysis would be used.

• uid - This is a simple library to deal with more performant and 'typesafe' UUIDs.

• memoization - This modules the ability to cache function results.

• extensions - General-purpose extension functions that can make your code more concise and readable.

Use Maven to run the unit tests as follows:

mvn clean test

To build the library JAR:

mvn clean package

If you wish to contribute to this project, please feel free to do so and send us pull requests. The source code formatting must adhere to the standard Kotlin formatting rules.

If you use IntelliJ IDEA, you can easily import the default Kotlin formatting rules like this:

Preferences -> Editor -> Code Style -> Kotlin -> (Scheme) Set From...

And then choose the pre-defined style Kotlin Style Guide. Voila!

Trace events offer a flexible way of logging semantic events during the run of an application. Semantic events are events that have some "external meaning", like "a route is planned", "the radio was switched on", "a connection to a phone was established", "a user logged in" and so on. (In contrast to debug messages likes "string is too long", "found record in database", etc. which only have an internal meaning to the program.)

The purpose of logging such events can be multiple, such as:

• for logging, or live monitoring of the system, when events are shown on a console or dashboard;
• for post-mortem debugging, to see what has happened in the system, after a bug was found;
• for system testing, where the test framework checks received events;
• for simulation purposes, where the events are forwarded to and processed by another system;
• to gather user behavior data, where user events are sent to a statistics module.

By default, a simple event logger is provided which logs all events directly to println. Although it is enabled by default, you can turn it off, or replace it with your own. It's easy to modify this to, for example, use the standard AndroidLog logger.

What makes this event logger different from a normal Log or, for example, Android Log directly, is that the events are 'type-safe': they are defined as Kotlin functions with (type-safe) parameters, rather than as handcrafted (and often refactor-unsafe) strings.

This means that rather than calling something like Log.d(TAG, "User logged in, user=$user"), you would write a type-safe expression like tracer.userLoggedIn(user).

The code for that looks like this:

(1)  |  interface MyTraceEvents : TraceEventListener {
(1)  |     fun userLoggedIn(user: User)
(1)  |  }
     |
     |  fun someFunctionToLogIn(user: User) {
     |      // Throw the trace event:
(2)  |     tracer.userLoggedIn(user)
     |  }
     |
     |  companion object {
(3)  |      private val tracer = Tracer.Factory.create<MyTraceEvents>()
     |  }

The lines marked (1) define the trace events; the line marked (2) throws (or logs) the event. That's it. Almost.

Trace events are defined in a local interface that derives from TraceEventListener. The interface lists all events as type-safe function calls. In order to get an object to actually be able to call these functions, you need to call Tracer.Factory.create from your class's companion object. This explains line (3).

It creates a proxy object tracer (also shown in line (2) above), which implements all trace events for you. The implementation of the trace event methods sends the serialized functional name and its parameters to an event queue. The events on this queue are subsequently consumed by trace events consumers, asynchronously. You can add a 'tagging' object to create. The class name of the object (typically this) may be added to log if @TraceOptions(includeTaggingClass = true) is used.

By default, a logging trace event consumer is enabled, which sends the events to a Log method that send the message to stdout or can be redirected to, for example, Android Log. This consumer is a special case in that it is actually a synchronous consumer, to make sure the order of events in the log is consistent with the other log messages from your application.

The logger can be enabled or disabled using Tracer.enableTraceEventLogging. It can also be switched to asynchronous mode. using Tracer.setTraceEventLoggingMode.

Note that only the logging consumer can be synchronous or asynchronous; custom event consumers are always processed in asynchronous mode.

You can replace the default logger, which logs to println with your own by creating an instance of the interface Log.Logger like this:

import android.util.Log
import TraceLog.Logger

object MyAndroidLogger : Logger {
    override fun log(logLevel: LogLevel, tag: String?, message: String, e: Throwable?) =
        when (level) {
            LogLevel.DEBUG -> Log.d(tag, message, e)
            LogLevel.INFO -> Log.d(tag, message, e)
            ...
        }
}

setLogger(MyAndroidLogger) 

Or, to reset it to the default implementation:

setLogger() 

Trace events in a TraceEventListener can be annotated with the @TraceLogLevel annotation.

@TraceLoglevel allows you to specify a log level, using parameter logLevel that will be used to output the trace event to a standard logger using a specific log level, such as INFO, DEBUG , ERROR, etc.

@TraceOptions offers the option to specify logging a full stack trace of a logged exception, using the optional parameter includeExceptionStackTrace. This applies to the last argument of an event ( if it is a Throwable object). By default, a stack trace is included.

@TraceOptions also offers the option to specify logging the filename and line number of the caller of an event. By default, the caller source code location is not provided.

@TraceOptions offers the option to add the class passed to create() to the log message, or omit that. By default, the caller class is not included.

@TraceOptions offers the option to add the interface that defines the event to the log message, or omit that. By default, the event interface is not included.

Example:

interface MyEvents : TraceEventListener {

    @TraceLogLevel(LogLevel.ERROR, includeExceptionStackTrace = true, includeOwnerClass = true)
    fun foundAnException(e: Exception)
}

Trace events are used for production code. Test cases should not use trace events. Test cases should use the logging tools of the test framework, or simply the Android log, to show progress or states.

By default, trace events are logged at LogLevel.DEBUG level to the default logger, that can be redirected to, for example Android Log. If you prefer to have certain events logged at a different log level, you can specify the log level with an annotation from TraceLogLevel, like this:

interface MyTraceEvents : TraceEventListener {

   fun userLoggedIn()           // This events gets logged at the default DEBUG level.
   
   @TraceLogLevel(LogLevel.WARN)
   fun cannotAccessDatabase()   // This events gets logged at WARN level.
}

This paragraph describes some simple coding conventions to promote consistent declaration and usage of events.

In general, events are coupled to specific classes and as such it is advised to declare them inside the class file that uses the events. Place the interface declaration at the end of the class.

Tip: In IntelliJ or Android Studio you can easily view all trace event interfaces by clicking on any occurrence of TraceEventListener and pressing Ctrl-Alt-B (or Option-Cmd-B)

The following naming conventions apply to events function names in a TraceEventListener:

Tracing is used for ongoing states and events that have occurred. It's not used for actions. As a result, the "trace event" names specify an ongoing state description, or use past tense. They don't use present tense and are not imperative.

Examples event and state names:

• Correct (events): userLoggedIn(), connectionEstablished()
• Correct (states): serviceReady(), establishingConnection()

Examples of incorrect names:

• Wrong (imperative): logUserIn(), establishConnection()
• Wrong (wrong state name): serviceInReadyState(), startingToEstablishConnection(), serviceInReadyState()

Note that there's not always a crystal-clear distinction between what events and states are, but for the naming convention that doesn't matter.

Trace event arguments are type-safe. They don't need to be strings. In fact, it's often better to pass the original object, rather than some form of string representation of it, as the object itself provides more information to trace consumer.

So, rather than defining an error event like

fun connectionLost(message: String)

when you know the message comes from an exception, like connectionLost(exception.message) you better define the event as

fun connectionLost(reason: IOException)

and invoke it as connectionLost(exception).

Trace events should always avoid throwing unexpected NPE's. This might happen if your trace event has a signature like

fun connectionEstablished(channel: Channel)

and you find that sometimes you need to call the event as connectionEstablished(channel!!), rather than as connectionEstablished(channel).

If that happens, you should consider changing the event signature to

fun connectionEstablished(channel: Channel?)

(and the invocation to connectionEstablished(channel)), just to make sure the !! operator cannot throw a NPE.

Especially in cases where the arguments for an event come from Java libraries, where it's not always clear if the type is @Nonnull (or @NonNull) or @Nullable, care should be take to make the event interfaces NPE-safe.

There are 2 types of trace events consumers:

1. Generic trace event consumers, derived both from GenericTraceEventConsumer and from a TraceEventListener interface.

   These consumers receive every event thrown in the system. They receive the event information as part of their GenericTraceEventConsumer.consumeTraceEvent implementation. This method gets a TraceEvent parameter that contains all information about the trace event, including date/time of occurence, class name, context information (per tracer instance, or even per thread the trace was created on), and the values of the parameters of the trace event method call (which can be obtained positionally, or by name from getNamedParametersMap, as a convenience).

   Generic consumers typically forward events to another system, such as the Android Log, store them in a database, or perhaps even send them across application (or machine) boundaries.

2. Specific trace event consumers, that implement a specific TraceEventListener interface.

   For example, you could implement the MyTraceEvents interface (see above) in a class called MyTraceEventsConsumer and register it as a trace events consumer. From then on, whenever a function from the MyTraceEvents interface is called, the corresponding implementation in MyTraceEventsConsumer will be called (asynchronously).

   Specific consumers typically provide specific handler code for specific events. They react on specific events, rather than forward them. For example, switching on a red light on an alarm dashboard, when the event temperatureTooHigh() is received.

Note that trace event consumer may receive mutable objects in an event, which may have been modified after the actual event was thrown. For more info, see the FAQ below.

The Tracer class defines the standard Log functions v(), d(), i(), w() and e(). These can be used like this:

tracer.d("Found record $record")    // Equivalent of `Log.d(TAG, msg)
tracer.w("Exception found", e)      // Equivalent of `Log.w(TAG, msg, e)

By default, tracers are set up to send these log messages synchronously to the default logger.

It is possible to register trace event consumers to process these log messages as well, for example, to send logs to another system for analysis.

If you wish to define a tracer to only log standard messages using d() etc., you don't need to create (or use) an TraceEventListener interface. You can just use this:

val tracer = Tracer.Factory.createLoggerOnly()

Note: Using the Log.x functions on tracers defeats the advantages of using type-safe arguments. Always consider using trace event functions, when possible.

By default, trace event arguments are formatted using their default toString function. If you wish to override them, use Tracer.registerToString<T>{ <toString implementation> } to override the toString method. For example:

Tracer.registerToString<Coordinate>{ "($lat, $lon" }

By default, the toString for Arrays is replaced with one that provides a list of elements, rather than an object reference.

Use Tracer.resetToDefaults() to de-register all custom toString handlers.

Sometimes multiple tracers may exist for a single class (if multiple instances of the tracer are initiated). In those cases, it may be necessary to be able to disambiguate the tracer that the trace events came from. This is solved by adding a context string to the create method. This context string is passed to trace event consumers. Alternatively, trace event consumers can specify a regular expression to make sure they only get the trace events for the specified tracer context(s).

// Declare 2 tracers.
val tracerMain = Tracer.Factory.create<SomeClass>(this::class, "main loop")
val tracerSec  = Tracer.Factory.create<SomeClass>(this::class, "secondary")

// Declare a consumer for events from any tracer starting with "main".
val consumerMain = MyEventConsumerForSomeClass()
Tracer.addTraceEventConsumer(consumerMain, Regex("main.*"));

// Declare a consumer for all events (leaving out the regex).
val consumerAll = MyEventConsumerForSomeClass()
Tracer.addTraceEventConsumer(consumerAll);

Note that only GenericTraceEventConsumers are able to retrieve the context string passed by the tracer (as it is part of the TraceEvent data object. Specific TraceEventConsumers (that implement the original tracer interface), cannot access the context while processing events.

Sometimes it's useful to be able to retrieve more dynamic information from a trace event that perhaps wasn't even available at the location where the event was generated, but at an earlier time. This effectively allows the developer to create much richer events than are normally possible.

Java logging introduced MDC, mapped diagnostic context, for that. This allowed Java users to add thread-local data that was added to log lines. The tracevents module allows something similar with the TraceThreadLocalContext feature.

Here's an example of storing additional context in trace events.

TraceThreadLocalContext.put("currentRequest", myRequest)

This would add a map called traceThreadLocalContext to the event with value {"currentRequest": "myRequest"} to each trace event generated from the same thread as this call to put was on.

The map traceThreadLocalContext is available for GenericTraceEventConsumers only and it is null when no thread-local data was stored, or a non-empty map when data was stored.

This is particularly useful for adding information that isn't necessarily available at the location where the trace event is generated. Without this feature, such events would require more parameters, just for the sake of adding context to events, which would clutter the main code.

For example, suppose you've written a media player and wish to generate a skipToNext event when the user presses the "skip to next" button. You would like to know the current song name when the user presses the button, but that information is not available in the "skip to next" method.

Let's have a look at what this looks like without the TraceThreadLocalContext.

Let's assume the media player always plays random songs and when you press "skip to next" it selects a new random song. The interface for the events for the media player could look like:

interface MediaPlayerEvents : TraceEventListener {
    fun eventPlaySong(songName: String)
    fun eventSkipToNext()
}

And the events to log the song name, the playing position, and the skip to next event, would be generated in different methods:

fun playRandomSong() {
    val song = chooseRandomSong(allSongs)
    eventPlaySong(song.name)    // <-- generate an event with the current song name
    startPlaying(song)
}

fun userPressedNext() {
    eventSkipToNext()           // <-- generate the sip to next event
    playRandomSong()
}

When this code is run, you'll get events like:

eventPlaySong(songName = "Dancing Queen")
eventPlaySong(songName = "Bohemian Rhapsody")
eventSkipToNext()
eventPlaySong(songName = "Breakfast in America")

Especially when these events are interlaced with other events, it becomes really hard to figure out which song was playing then "skip to next" was pressed. And if the program is multi-threaded, it becomes undoable.

Now, let's have a look at the code when TraceThreadLocalContext is used. Eveything remains the same except for the implementation of playRandomSong:

fun playRandomSong() {
    val song = chooseRandomSong(allSongs)
    TraceThreadLocalContext.put("songName", song.name) // <-- store the song name in (thread-local) data
    eventPlaySong(song.name)                          // <-- generate an event with the current song name
    startPlaying(song)
}

The events that are generated now, look like this:

eventPlaySong(songName = "Dancing Queen", traceThreadLocalContext = null)
eventPlaySong(songName = "Bohemian Rhapsody", traceThreadLocalContext = {"songName": "Dancing Queen"})
eventSkipToNext(traceThreadLocalContext = {"songName": "Bohemian Rhapsody"})
eventPlaySong(songName = "Breakfast in America", traceThreadLocalContext = {"songName": "Bohemian Rhapsody"})

Now it becomes trivial to see that "Bohemian Rhapsody" was skipped, and that "Breakfast in America" was played before that.

Advanced examples of using this trace event mechanism are:

• Sending events to a simulation system, which simulates the environment of the system. For example, an event that the cabin temperature has been set, may be processed by a simulator which uses a trace event consumer to receive such messages.

• Displaying events on a dashboard, to gain more insight in the current status of the system, rather than having only a scrolling log to look at.

• Collecting or sending system usage data for analytics. Developers can define all sorts of semantic events, and the system may collect them easily in a database, for later processing.

• Should I use Trace or (Android) Log?

  As a rule of thumb: you should always use Tracer instead of Android Log. If you only want to regular log messages, you can use the log functions d() etc.

• What is the performance penalty of using Tracer over Android Log?

  Using the default (synchronous) logging of log messages via Tracer, over using Android Log, introduces a performance overhead of less than 20%.

• Is it safe to trace mutable objects?

  Yes and no. The tracer module never modifies objects. But consumers of trace events will receive a mutable object, which may have been modified since the event was thrown. This may not be expected by the event consumer, although often it's not a problem at all. If you have to get around this you need provide a clone of the object in the trace call yourself. Beware that cloning objects to make them immutable is relatively expensive and may generate significant overhead.

For more information, for example on how to define and register trace event consumers, please refer to the documentation in the Tracer class.

Generic immutable unique ID class. Really just an abstraction of UUIDs. Used to uniquely identify things. No 2 Uid objects shall be equal unless they are the same instance of the Uid class or their underlying UUIDs are equal.

The class has a generic type T to allow creating typesafe Uid's, like Uid<Message>. The class represents UUIDs as Strings internally, to avoid loads of UUID to String conversions all the time. This makes the class considerably faster in use than the regular Java UUID class.

Examples:

var messageId = Uid<Message>()    // Creates a new unique message ID.
var personId = Uid<Person>()      // Creates a new unique person ID.

messageId = personId              // <-- Does not compile, the IDs are type safe.

val testId = Uid.fromString("1-2-3-4-5")    // Allows shorthand notation for UUIDs,
                                            // easy for testing, or input.

val s = messageId.toString()      // Serialized to string.
val id = Uid<Message>(s)          // Deserialized from string. Faster than `fromString`
                                  // if the format is known to be the serialized format.

val messageId: Uid<Message>               // If you need, you can translate IDs from one type to
val personId = messageId as Uid<Person>   // another using 'as'. This is useful if the type 
                                          // information was lost, for example, in serialization.

Provides the memoize extension to Kotlin functions that allows optimizing expensive functions by caching the results corresponding to some set of specific inputs.

In order for memoization to work properly:

• all input arguments must have proper equals and hashCode implementations,
• given the same input, the function must always return same output (i.e. not dependent on external parameters), and
• the function should not exhibit any side effects, other than the returned result.

To memoize a function, use the extension:

• memoize(Int) - Extension creates Least Recently Used (LRU) cached function that will remove least recently used item if number of stored results exceeds provided limit.

Note that currently only functions with a maximum of 4 parameters are supported by memoize.

Example:

val function1: (Int) -> String = { p1: Int -> p1.toString() }.memoize()

function1(10) // First call with 10 - actual function is called and result is cached.
function1(11) // First call with 11 - actual function is called and result is cached.
function1(10) // Second call with 10 - value is returned from cache.

General-purpose extension functions that can make your code more concise and readable. These extension functions aim to compliment Kotlin's functional programming style where Kotlin's default syntax and stdlib requires unwanted boilerplate code.

For example, it allows replacing the following snippet:

someComponent.getSomeNullableValue()
    .let { it ?: someComponent.getFallbackValue() }
    ?.let {
        if (it) doSomething()
        else null
    }

with:

someComponent.getSomeNullableValue()
    .ifNull { someComponent.getFallbackValue() }
    .ifTrue { doSomething() }

Copyright (C) 2020-2022, TomTom (http://tomtom.com). Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

[http://www.apache.org/licenses/LICENSE-2.0]

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Use Maven to run the unit tests as follows:

mvn clean test

To build the library JAR:

mvn clean package

If you wish to contribute to this project, please feel free to do so and send us pull requests. The source code formatting must adhere to the standard Kotlin formatting rules.

If you use IntelliJ IDEA, you can easily import the default Kotlin formatting rules like this:

Preferences -> Editor -> Code Style -> Kotlin -> (Scheme) Set From...

And then choose the pre-defined style Kotlin Style Guide. Voila!

Author: Rijn Buve

Contributors: Timon Kanters, Jeroen Erik Jensen, Krzysztof Karczewski

• Dependency updates in pom.xml. No code changes.

• Introduce caching of trace event method annotations in the Tracer.invoke() function.
• Skip creation of a TraceEvent object if there are no event consumers. This saves us from performing an expensive LocalDateTime.now() call.
• Skip retrieval of trace event method annotations for simple log function invocations, if there are no event consumers.
• Remove string concatenations in createLogMessage() functions. Only use StringBuilder functions.
• Change toStringRegistry to use item::javaClass.name as key, and use item::class.toString() only as fallback.
• Skip reading item.javaClass.getMethod("toString").declaringClass in convertToStringUsingRegistry(): the item.toString() function of Any will anyway print the name of the class.
• Fix execution of unit tests requiring mockk-jvm (extensions and traceevents modules): no unit test for these modules was being run anymore, after the 1.8.2 update.
• Fix the IfExtensionsTest unit tests: these tests were broken by mockk 1.13.4 (due to mockk/mockk#1033).

• Dependencies updated.

• Events traces are not queued till the processing starts.

• Added Tracer.hasQueuedTraceEvents: Boolean to support testing that certain trace events were not sent.

• Fixed documentation on logging sync instead of async by default. No functional change.

• Remove use of reflection on the critical path of creating a Tracer object by removing the use of KClass<*>.isCompanion. Remove the $Companion suffix from the tagging class instead.
• Initialize Tracer library faster by removing the use of reflection to obtain Tracer library class names.
• Replace the use of LocalDateTime with Instant to measure elapsed time. This omits loading timezone data during initialization of the Tracer library.
• Improve documentation of TracerEvent.parameterNamesProvider.

• Reduce use of reflection on the critical path of trace events by not obtaining the trace event parameter names on the thread that traces the event. Only obtain the trace event parameter names when requested by a trace event consumer.

• Updated dependencies.

• Added Kotlin Contracts to ifTrue and ifNull.

• Removed cast extensions.

• Enables explicit API, requiring stricter type and visibility definitions of our public APIs.

• Added extensions module, providing general-purpose extension functions that can make your code more concise and readable.

• Updated dependencies.

• Added getNamedParametersMap to TraceEvent, which allows generic trace event consumers to access parameters and their values through a map that maps parameters names to their values. The other way to access parameter values is by using args, which is an array with parameter values, in the order of the method declaration.

• Added ability to store thread-local (MDC-style) context to trace events using TraceThreadLocalContext. This context can be processed by GenericTraceEventConsumers. Initial idea by Chris Owen.

• Updated POM dependencies.

• Replaced deprecated Channel methods offer and poll with current method variants trySend and tryReceive.

• Replaced some internal properties with private to be stricter on visibility.

• Removed redundant suspend modifiers from methods.

• Updated POM dependencies.

• Removed dependency on kotlin.bintray.com.

• Added context to logging, if context is not empty.

• Added context to Tracer.create to allow disambiguation of tracers, if there are multiple for the same class.

• Updated dependencies and copyright.

• Added Kotlin function memoization extensions.

• Updated dependencies (except mockk as 1.10.2 will fail the test for unclear reasons).

• Added Uid class for UUID handling.

• Minor bug fixes.

• Updated dependencies. No functional changes.

• Made TraceEventConsumerCollection a public class to be able to use it also for consuming streamed events.

• Minor bug fixes.

• Made TraceEvent.stackTraceHolder nullable.

• Bug fix, renaming non-DEX formattable function name.

• Renamed annotations: @TraceOptions(includeExceptionStackTrace, includeTaggingClass, includeFileLocation, includeEventInterface)

• Added ability to add tagging class to tracers.

• Removed restriction that tracers can only be created in a companion object.

• Cleaned up annotation @TraceLogLevel to only include trace level.

Added @TraceOptions(includeExceptionStackTrace, includeCalledFromClass, includeCalledFromFile, includeEventInterface)

• Added throwableHolder to TraceEvent so event handlers can inspect the stack as well.

• Added includeFileLocation to @TraceLoglevel add the caller filename and line number to the logger.

• Added includeExceptionStackTrace to annotation @TraceLogLevel.

• Added includeOwnerClass to annotation @TraceLogLevel.

• Removed time stamp from SYNC logging (already added by most loggers).

• Added unit tests to check message formats.

• Added Tracer.Factory.createLoggerOnly(this).

• Rename Log to TraceLog and LogLevel to TraceLogLevel.

• Fixed string representation of arrays.

• Added unit tests for message formatting.

• Added Tracer.RegisterToString to register string handlers for class types.

• Fixed license and copyright messages to Apache License 2.0.

• Bug fixes to simple loggers Tracer.d(message, exception).

• Fixed formatting of event parameters for arrays and lists.

• Fixed unit test helper method.

• TAG is non-nullable for loggers.

• Renamed directory structure from src/main/kotlin to /src/main/java for IntelliJ to understand package names

• Added TravisCI support for Github, including status badges in README.

• Initial release