Async I/O

[danielhuang]

I wonder if async I/O would offer a performance benefit. Right now, the filesystem operations are always blocking, even if performed using tokio. However, if the target is accessed via network, or if the recent io_uring interface is used, async performance provide a benefit.

[sourcefrog]

I look at this as a few levels:

First, get more overlapped IO altogether. There's a lot of scope for more of this. I think this is almost certainly the largest available win.

Secondly, should this be via blocking syscalls from multiple threads (as at present) or io_uring or something else.

Third, should it be in userspace as threads or Tokio or something else.

In Conserve there is a mix of IO and userspace CPU for hashing and compression; the mix will depend on the tree and the relative speed of the CPU vs IO. On flash, on a backup with many changes, there can be lots of cycles in userspace. So it's important to always keep NCPUs threads full. We do not yet already get there.

If many threads issue blocking syscalls that is probably pretty fast already; possibly enough to keep an SSD busy. io_uring might manage to issue more IOs, but at the cost of entangling with an interface for which I think there is not yet a clean cross-platform abstraction, and which is not yet universally available on Linux. So, it's really interesting, but I would not yet set a direction to migrate to it.

Given that we're doing a lot of userspace computation across O(NCPU) threads and local IO would Tokio help? My impression is that it's not great at that scenario, and it's more targeted at >>NCPU tasks doing network IO. Does it help enough to be worth any runtime overhead or development complexity? My impression from experiment is it does not. For example it seems to break local IO into small ~8kiB blocks, but we almost certainly prefer very large single reads. It does not seem to try to minimize syscalls.

So, that's what I think at the moment...