Fix tracking run inputs

[Zethson]

We observed that when data from any non-default instance such as cellxgene is used as input during a Run, it does not show up in the linage graph.

@lazappi had suggested that this is probably because of our usage of the API to get the data and not the Python code which probably does more magic.

[falexwolf]

The problem is that reticulate isn't used for .load() and .cache().

If reticulate was used, this would all be resolved.

Hence, the fix should be using reticulate for these two methods and it's going to work.

[lazappi]

I think we should have a discussion about whether it is worth using the API at all. We have about reached the limit of what the API can do currently and if we have to use {reticulate} for some things anyway, maybe it's better to use it for everything? It would mean a fairly big refactor but after that development might be quicker.

[falexwolf]

Yes, we should have that discussion next week or so, I agree.

But what's indeed much better with the API is that you're not relying on 20 Python packages that Django needs to map all the schema modules.

So, it's not a clear decision pro reticulate for querying say bionty. That's likely more elegant through the REST API.

[falexwolf]

Under the hood of artifact$load() one calls:

def load():
     uid = from REST
     artifact = ln.Artifact.get(uid)
     return artifact.load()

[rcannood]

Solution 1

• Add a rest endpoint for letting the lamindb instance know that we are caching a certain artifact in the context of a certain transform

Solution 2

• Switch to reticulate for caching files

Solution 3

• Use rest API as long as db$track() has not been called, switch to reticulate once tracking has started.

[falexwolf]

I'm strongly favoring Solution 2 for the up-coming weeks.

[falexwolf]

@lazappi (or @rcannood), do you have time to fix the this Monday morning so that this goes into the 0.3.0 release until about 2 pm?

I believe nobody really tried laminr much last week because of Thanksgiving. But they will this week.

Having tracking of inputs is the only missing key piece.

[lazappi]

I'll have a look this morning. There are other things I would like to improve but this one we haven't looked at yet so I'll do it first.