Evaluating JuiceFS with Litestream

[stevefan1999-personal]

I was curious about how can we re-utilize the S3 object storage server to use it as an metadata server (i.e. S3FS), this will not only save a lot more on database cost but also backup cost. Then I came across Litestream after this Tailscale post (in fact I met this multiple times before but it was not yet mature, although it isn't much better now either). So it got me thinking we could use JuiceFS as standalone mode, and then do SQLite replication for the metadata to S3 with Litestream, we could have solved this age-long problem.

The achilles' heel here is of course Litestream does not support multi-HA and that S3 only provides eventual consistency, so you are virtually unscalable. But for a small user like me who cannot use S3 as merely an object storage (and more akin to NFS) this would be fantastic. Wonder if someone also did the same thing?

[solracsf]

See also #1475

[davies]

When JuiceFS is not accessed concurrently from multiple servers, Litestream could be a better option than SQLite alone.

The difficulty will be how to make sure that JuiceFS is only accessed from on server, otherwise it will be corrupt and lose data.