Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling of multiple connections per peer #151

Open
madadam opened this issue Jan 16, 2024 · 0 comments
Open

Improve handling of multiple connections per peer #151

madadam opened this issue Jan 16, 2024 · 0 comments

Comments

@madadam
Copy link
Collaborator

madadam commented Jan 16, 2024

We can have multiple connection for the same peer. The way this is handle currently is that we multiplex those connections onto one "virtual connection" which we then treat as if it were a single connection. In details, incoming messages are multiplexed onto the virtual connection (that is, all messages from all connections are received) but outgoing messages are sent only on one connection - the first one in the list of connections. If the sending fails, that connection is removed from the list and the send is repeated on the next connection. This means that in practice only one connection per peer is actually being used at any given time.

This design is not ideal for several reasons:

  1. Even though only one connection is used, the remaining connections are still kept open using keep-alive packets which can contribute to battery drain (although the keep-alives are sent once a minute which might not be too bad in practice).
  2. The active connection is picked arbitrarily regardless of how "good" that connection is. For example, consider a peer we have two connections to - one local and one global. The current system might blindly use the global one even though the local one has likely better bandwidth.
  3. The implementation is too complex (for example, it requires complicated components like MultiStream, MultiSink or Barrier to work) and hard to reason about and debug.

Proposed improvements:

Instead of keeping multiple connections per peer, we keep only one, but have a mechanism to replace it with another connection if it is "better". In more details:

When a new connection is established, we perform the handshake. If it succeeds, we obtain the peer's runtime id and also it proves the connection is good. We then check whether we already have a connection to that peer. If not, we set it as the active connection for that peer. Otherwise we compare the existing and the new connections to determine which is "better" (more on that later). If the existing one is better, we close the new one but wait until the existing one is closed and then try to re-establish it. If the new one is better, we close the old one and replace it with the new one.

In order for this to work consistently, only one of the peers will perform the connection selection. We need some way to pick this peer which both peers would agree on. One simple way which might be sufficient is to pick the one with the higher runtime id.

Determining which connection is better can be done in multitude of ways. One very simple one is to consider only the connection protocol (TCP, QUIC, ...) and IP address:

  1. Prefer local over global
  2. Prefer QUIC over TCP
  3. If both local (or global) and both have the same protocol, prefer the existing one

A more robust way would be to measure the round-trip time of the connections and pick the lower one. For QUIC, the quinn crate exposes an API for that. For TCP this info exists (linux, windows) as well, but there is currently no API in rust (tokio or std) to expose it. It might be possible to expose it ourselves but it would require writing a low-level and very platform dependent code. Alternatively, we can measure RTT ourselves at the application level, or use the RTT test only for QUIC connections.

That said, the simple protocol + ip based algorithm would likely already be an improvement over what we have currently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant