Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement request] Publish logs to Elastic DataStream #948

Open
patpatpat123 opened this issue Apr 19, 2023 · 0 comments
Open

[Enhancement request] Publish logs to Elastic DataStream #948

patpatpat123 opened this issue Apr 19, 2023 · 0 comments

Comments

@patpatpat123
Copy link

Hello team,

I would like to reach out with a small enhancement request, please.

Currently, this logstash logback encoder is doing a great job forwarding application logs to Logstash (which will then send them to ElasticSearch).
However, in ElasticSearch, they live inside a regular index.

Would it be possible from this project to send the logs to ElasticSearch, but to a DataStream, not just a regular index:
https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html

(Copy paste from ElasticSearch official doc:)
A data stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Data streams are well-suited for logs, events, metrics, and other continuously generated data.

(see screenshot for datastream)
Screenshot 2023-04-11 163729

Currently, this is only create a plain index (please see screenshot)

Screenshot 2023-04-11 163814

Rationale
The reason behind this request is that by definition, logs are very well suited for ElasticSearch DataStream, because they are data stream in the first place.

However, as of now, one needs to manually convert the log index into DataStream from the regular Index on the ElasticSearch backend in order to benefit from the synergy between this encoder and elasticsearch data stream.

Would be a great feature if this encoder can directly push the logs as it is doing today, directly to a data stream instead.

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants