Use tar archival format to transfer file tree structure and data between Polykey vaults and local file systems

[aryanjassal]

Specification

There are times when we need to transfer the secrets from a vault to either another vault or the user's file system. Sometimes, only one secret needs to be transferred. Other times, we need to transfer multiple file trees including their directory structure.

As all the vaults are stored on the same encrypted file system (efs), to transfer file trees between vaults, we only need to use regular file copying/moving operations on file systems; something along the lines of fs.promises.copy() should work well to transfer secrets between vaults.

However, doing this between the vaults and the user's file system is not as straightforward. To efficiently transmit the file tree, we will be using an archival format like tar. The tar archival format is inherently streamable, and can be used to zip the file tree into a single file, which can then be transmitted over a RPC call, then be unpacked on the client, effectively transferring the file structure to the user's file system. Of course, we can also compress the resulting file, but we won't get into that quite yet.

Additional context

• Pokykey#799 has seen discussion regarding streaming over file trees using RPC calls.
• gera2ld/tarjs can be used as a zero-dependency package to generate tar from a file system.
• matrixai/js-virtualtar can be looked into, as this was also attempting to make streamable tar bindings for JavaScript.

Tasks

1. Make an RPC handler responsible of copying/moving file tree.
2. To move file tree between vaults, just use the fs operations. Multiple locks might be required if transferring between multiple vaults.
3. To move file tree between vaults and file systems, make a tarball and stream it over RPC instead.

[linear]

ENG-412 Use `tar` archival format to transfer file tree structure and data between Polykey vaults and user file systems

[aryanjassal]

After taking a look at matrixai/js-virtualtar, I have realised that significant work still needs to go into the repo before it would be ready to be used within the Polykey ecosystem. This alone might take a cycle or two. I will need to play around with it before starting development on it.

[tegefaulkes]

What needs to be done before it's ready to be used?

[aryanjassal]

I haven't taken an in-depth dive into the requirements, just a brief skim-through over the code. As such, I'm not sure what exactly needs to be done for that. I'll need to see what it already has and what still needs to be done when I start working on this issue.