-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
async redis cluster should use initial startup nodes during reinitialization in case of failover #2472
Comments
I'll be happy to review any PRs with a fix. |
Hi! Is there any update on this? I got the same issue when I use AWS MemoryDB.
I thought doing some reinitialize would be working, but it was not. |
Hi! I got the same issue too when I use AWS redis. Exactly the same exception as Jiyeonseo posted:
Is there any updates on this issue? Thanks! |
Heads up that the "this line" link in the original code description pointed at Given that state of the repository at the time the issue was created, the line in question is redis-py/redis/asyncio/cluster.py Line 1337 in d3a3ada
|
Seems like adding |
Version:
redis-py: 4.3.3
redis: 7.0.2
Platform:
python 3.7
Description:
Async implementation of
RedisCluster
overwrites initial startup nodes during first initialization. This is caused by this line.When deployed in dynamic systems like Kubernetes your shards may sometimes change their IP addresses.
This likely leads to a situation when python client ends up with a pool of invalid (outdated) startup nodes and can not perform reinitialization, since
NodeManager
uses raw IP addresses.For example, let's say we have cluster with 3 masters in k8s and use python client this way:
now image your python client is idle for some time without requests, and during that period K8S have recreated each redis container due to whatever reason (crash, or moved to another node by autoscaler for example).
Then, next time python client will try to perform a redis call it will find out that all ClusterNodes are unavaliable, will try to reinitialize NodeManager and end up with no connectable node to get new cluster configuration from.
This would not be an issue if initial headless URL was still in the pool of startup nodes.
Sync implementation of
RedisCluster
preserves initial node address just as suggested above, so i believe async implementation should behave the same.For now I use the following patch, but I can't call it an elegant way to get around the problem. It must be solved through changes to NodeManager.
P.S. sync implementation has the opposite problem,
NodesManager
doesn't pop unused addresses from startup_nodes storage => it may grow indefinitely as soon as your containers change IPs.The text was updated successfully, but these errors were encountered: