Confused about importing Elastic index into Timesketch via the API

[RC810]

Hi there,

I've read through the provided documentation and it feels very sparse in this regard. The situation is like so:

• I have timesketch running on its own VM
• Elasticsearch is running on its own VM as well
• I'm using a 3rd VM to do processing and this is where log2timeline is as well as the timesketch_importer
• I run log2timeline and then psort against my collection to push the data into Elasticsearch in timesketch format (using the -o elastic_ts flag on psort)

This all goes without a hitch, however I have the following issues:

1. Once the data is ingested into Elasticsearch, I can't see it in Timesketch and I can't seem to figure out how to make it available in Timesketch. There's no option from the GUI to bring ingested data in, so I'd have to rely purely on the CLI and... I'm confused about how I would take an existing index and add it to Timesketch. This would be my preferred way
2. I tried the timesketch_importer, but it seems to fail with an error regarding connection issues even though I'm pointing it at the correct IP address in the local network (I verified that the machines can all communicate with each other and that the Elastic server is accessible in the VM).

So, all said and done, I'd really like some help trying to figure out how to do 1. I'm not a python dev and I'm not used to working with API's, so I'd appreciate any help that can be provided in trying to get this to work.

Thanks!

[jaegeral]

Hm the preferred way should be the timesketch_importer. Are you sure that the system you run it on has connection to the TS backend? Can you open a netcat in the same direction?

[RC810]

Thanks for the reply! I had kinda given up hope on getting a response here if I'm honest haha. I kept researching and experimenting and was able to get timesketch_importer to work, though with some caveats. It's really picky about specifying the index name and it would always fail to upload plaso files if I included that. I have to let Timesketch handle the index creation (even if I've made an index beforehand) or it keeps failing to upload. That was one of the issues I was having that was really confusing.

One particular issue with doing Log2timeline > timesketch_importer > timesketch > Elasticsearch is that there's really no place for Logstash in there and so duplicate events can get indexed. There's also no way to filter out duplicate events using timesketch_importer right now (i.e. Windows event logs creation/modification time) so they have to be indexed first and then purged afterwards via a curl command. It's kinda janky, but it works.

The reason I was thinking Log2timeline > Logstash > Elasticsearch > Timesketch would be better is because all of those duplicate events could be figured out, thus minimizing the index size and speeding up the whole process. Then the API client can be used to create a sketch and make the data available within Timesketch for analysis.

But I'll be honest... I don't really understand the API client. I figured out how to create a sketch thanks to the examples posted, but I can't even figure out how to delete a sketch via the API haha. I don't even know how I'd do the above via the API client since I feel there's virtually no documentation surrounding that (aside from one very vague code example at the bottom of one of the developer pages). I would greatly appreciate more insight into that or at least some more robust examples since my understanding of API's and Python especially is just not that strong.

Thanks!

[jaegeral]

Hm multiple things here:
Log2timeline > timesketch_importer > timesketch > Elasticsearch is that there's really no place for Logstash correct.

The "duplicate" events you mentioned afaik are not really duplicate events, but two distinct events. They might not add value to your analysis workflows, but might for others.

Timesketch is really not made to work on data that has been added to ES in other means than the timesketch import client, the API client or web upload. The direkt import to ES might miss some metadata that we store alongside to the ES sketch, ACLs and many more. It might work, it might not work, it might break your installation in the future, we don't know and do not recommend doing that.

As for the API client, documentation can always be better, agree. Maybe it is worth to have a look at https://github.com/google/timesketch/tree/master/notebooks where each notebook makes heave use of the API client.
Covering the flow you explained above with the API client sounds partially possible, but adding a sketch with already existing ES data is not in scope, so pretty sure there would be rough edges. Same goes for deleting events, it is nothing that is a main concern to the functionality of Timesketch, as the evidence added should (in most cases) be treated as one piece of evidence, removing one or more events from the evidence can cause problems, which is why we designed the system to make it non trivial to do that. You can hide events in the UI, you could mass hide events using the API (or it should be trivial to implement if it is not already there), but deleting events might be harder to do.