Network partition #304
Replies: 7 comments 13 replies
-
|
This is the doc about how Olric architecture: https://github.com/buraksezer/olric?tab=readme-ov-file#architecture. I have not done such experiment yet to see how olric handles such situation. |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll this is how network partition is handled by Olric https://github.com/buraksezer/olric?tab=readme-ov-file#pacelc-theorem |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll adding here the link to the issue #290. |
Beta Was this translation helpful? Give feedback.
-
There is a |
Beta Was this translation helpful? Give feedback.
-
|
The Rejoin has never been used by the way. |
Beta Was this translation helpful? Give feedback.
-
|
There is one experiment I did and this the scenario:
The cluster seemed disrupted because I could only access AA from A and BB from B. If I tried accessing AA from B after killing node C it could not find it. The same for BB. I am trying to understand a bit the behavior from the Olric code and doc. |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll you can reproduce this using the sample cluster app here https://github.com/Tochemey/goakt/tree/main/examples/actor-cluster/dnssd. |
Beta Was this translation helpful? Give feedback.
-
|
@Tochemey Well then it is another problem. What I mean is that if you cut off the connection between C and (A, B) for long enough time without killing C. C will be happy to be a one node cluster. I really think the algorithm used by Orleans is excellent but there is no golang implementation. It is even better with a storage-based implementation of grain directory. I am trying to implement this in golang with storing MemberShipTable and grain info in etcd all together but it is way more difficult than I think. |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll I have been thinking moving away from Olric. I have been looking at using groupcache and some network partition handling on top of it. Another thought is to use raft+membership to build some distributed KV store. There are jus thoughts at the moment. Nothing concrete yet. What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
@Tochemey I personally prefer the latter one. It can achieve stronger single-activation guarantee. I have been thinking using etcd for service discovery and split-brain resolution and using some gossip protocol like memberlist for direct failure detection. The downside is that it is not self-contained. I do not think embedding raft is a good idea becasue every node has to do heartbeating with every other node(correct me if I am wrong). |
Beta Was this translation helpful? Give feedback.
-
|
Maybe we can fork Olric and modify it to fit our needs :) |
Beta Was this translation helpful? Give feedback.
-
|
From my observation Olric does not resync keys amongst cluster nodes. |
Beta Was this translation helpful? Give feedback.
-
We can embed raft. I started in the early stage using the embedded etcd before I discovered Olric. And it was working well https://github.com/Tochemey/goakt/blob/v0.2.0/cluster/cluster.go. My challenge then I was not using the discovery well. |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll you can see the implementation here https://github.com/Tochemey/goakt/blob/v0.2.0/pkg/etcd/ Maybe we can switch back. |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll these were the caveats I faced then https://github.com/Tochemey/goakt/tree/v0.2.0?tab=readme-ov-file#operations-guide. And honestly the same applies to Olric as at now because a healthy cluster needs to have three nodes. |
Beta Was this translation helpful? Give feedback.
-
|
@Tochemey However, if etcd is used as an external dependency, there is no quotum requirement on working nodes. Only the health of the etcd cluster is needed and it can tolerent one failure at the scale of 3. Maybe you have already understanded what I mean. |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll I understand. I just wanted a self-contain system. Maybe I am wrong :) |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll I hope you are well. I would like to know whether after going through the code whether you have an idea on how to get this done. |
Beta Was this translation helpful? Give feedback.
-
|
Not really. Without introducing external dependencies, the best we can do is adding some algorithm to resolve this. |
Beta Was this translation helpful? Give feedback.
-
|
@qazwsxedckll let me do some research and see what comes out. Do you suggest we open an issue to proper tackle it? Would you mind adding some description when we create an issue out of the discussion? |
Beta Was this translation helpful? Give feedback.
-
Have you checked how olric handles network partition? I have done some experiments on hashicorp/memberlist. If I isolate some node from others, they will form a new cluster and never rejoin when the network is healed. It seams that
func (d *Discovery) Join() (int, error)is only called once on start. I want to know how olric can handle this situation.Beta Was this translation helpful? Give feedback.
All reactions