[Meta] Development plans

[Mpdreamz]

Hi Everyone!

@mivano and I met a couple of weeks ago to discuss the current state of the sink and where we would like to see the sink move towards to in the future.

• Define an EOL policy
  The sinks should follow Elastic EOL policy. This will mean dropping support for 2.x and 5.x.

• Integration Testing
  Started on this here using the same integration testing framework (based on xunit) that the Elasticsearch .NET clients use.

• Support for ILM
  Index Lifecycle Management is a great way to manage your indices and it'd be great if the sinks exposed ways to control what happens to older log indices.

• ECS
  Provide a formatter that plays nice with The Elastic Common Schema. Already requested for the sink and this is something that we at Elastic are exploring to expose to the .NET world as well as a dependency free nuget package.

• Move durable mode reponsibility
  Durable is a must have feature for sinks in general and so ideally this does not live in this sink but becomes a Serilog core responsibility. In the long run this sink should not take on the burden cc @nblumhardt

• Move .NET baseline
  The clients baseline is netstandard2.0 and net461 and so moving forward the sink will use this as the baseline for .NET Core and .NET Full Framework releases.

Nothing is set in stone, nothing has a timeline.

Opinions and feedback will be much appreciated!

[nblumhardt]

Awesome 👏

RE the future of durable mode, on the one hand it would be great to extract this somehow (we're maintaining the same code in a few other sinks), but on the other, I've been recommending against using it whenever I have the opportunity, since it's not generally a pit-of-success:

The implementation is okay in limited use cases, but as soon as multiple processes are involved (most IIS-hosted web apps, because of overlapped recycling) or when storage is ephemeral (most cloud-hosted web apps) the approach breaks down.

I don't think that rules out extracting it and working to improve it, but it might be worth breaking out another thread to analyze the current state of affairs and convince ourselves that we wouldn't be churning away at a dead end. What do you think?

[mivano]

Thanks @Mpdreamz for the summarization and the involvement!

@nblumhardt the durable mode has always been a little troublemaker. It works great in simple scenarios and is quickly enabled. But the features are limited, a lot of code duplication even in this sink alone and if you really want guaranteed delivery, then it will not be the best solution. Hence the reason to think about removing it and keep the sink a bit simpler.

A more generic solution where you can link sinks together based on conditions like

Log.Logger = new LoggerConfiguration()
    .WriteTo.Durable(w => w.Sink1(primary, ), durableOptions)
          .ThenTry(w => w.Sink2(secondary, ))
          .FallbackTo(w => w.File(secondary, ))
    .CreateLogger();

Might be something useful higher in the chain, but will require a better design and need.

[nblumhardt]

@mivano thanks for the reply, I think we're on the same page, then 👍

The main thing that has prevented me, in the past, from wanting to make it a generic component is the problem of what to do with Exception, which isn't necessarily round-trip-serializable to the buffer file.

Sinks that use durable mode seem mostly to be JSON-based, so a generic component based on writing and reading back lines of text would be closest to what we have here today, but hooking that up to WriteTo.Durable() would require sinks to expose some kind of batched-lines-of-text interface, which seems questionable.

The alternative would be to accept that we write exceptions out using ToString() to the buffer file, and read them back in using a surrogate Exception-derived type that wraps up the string representation of the exception, but sinks that treat exceptions as structured data wouldn't work well with this. (Even in this case, an IBatchedSink interface would need to be introduced.)

If we're just extracting WriteTo.Durable() in order to deprecate it, the second option (a surrogate Exception type) seems like a reasonable trade-off. If we're extracting it with the intention of improving and maintaining it then some deeper thinking is going to be needed.

🤔

[Mpdreamz]

I don't think that rules out extracting it and working to improve it, but it might be worth breaking out another thread to analyze the current state of affairs and convince ourselves that we wouldn't be churning away at a dead end. What do you think?

Huge fan of this idea, scoping the benefits and trade-offs of a Durable mode would be hugely beneficial. Also scoping durable in general, does it always mean buffer to file first or should it be open enough to allow for @mivano fallback approach or a combination of these?

[orjan]

@mivano what about rebooting this issue?
Many of the tasks mentioned here is still relevant and we're even approaching EOL of v6. Maybe it's time for outlining a plan for the next major version as well as a migration plan for teams moving from v6 => v7 => v8. I can image something like making sure we're having basic support for Elastic v8 in the current version, then it should be possible to upgrade fairly smooth. In the next version we'll might support for everything below v7?

[mivano]

We need indeed make some progress/plans here. Maintaining multiple versions and the buffer option are main issues of issues we see here. We should indeed stop the formal support of v6 (people can still use an older version of the sink). I m also inclined to remove the durable option; either completely or to at least a separate package if possible. At the same time, also align the supported frameworks. Both these things are pretty breaking for sure but to me acceptable as work around are available.

There is now also basic support for ECS and the ILM looks sensible for me as well.

I really welcome some help here as my primary usage of this sink is almost zero due to other work and projects.

My primary goal with this ES sink was always to have something that just works, so adding a one-liner should give you logging in ES. I believe we still have this, but also the legacy and code of the last x years. So it might be time to cut some features...

[vakuncha]

Can we add retry option for the batch Elastic sink?

[bbarry]

With regards to ECS, Perhaps a comment here or a note in the task above about how the https://github.com/elastic/ecs-dotnet project relates to this. Eg are we recommended to consider that sink instead of this one or is that a proof of concept type of thing or is there seriously a plan to indefinitely maintain both projects (which to someone just getting started with this appear to do the same thing)?

[mivano]

You can use this sink, but set the customFormatter to use Elastic.CommonSchema.Serilog.EcsTextFormatter instead. This will format it to the ECS format.