Failed to receive valid reponse after 3 retries #76
Comments
|
Hello, friend. I have the exact same problem that you described. Since December 22, from time to time, I receive this same message. Were you able to fix this problem or it just stopped? Thanks for your time. |
|
We're seeing this issue too! I've been asking Customer.io support about it. |
|
I thought it might be a thread safety issue, as we are running multithreaded environment and the CIO python client shares an instance of Also I noticed that only transactional email has this problem - |
|
Same here, the complete error message would be this , at least for me. |
|
We see this very regularly when using the Python SDK up to the point of it becoming unusable. This is what I believe is going on: The def __init__(self, key, url=None, region=Regions.US, retries=3, timeout=10, backoff_factor=0.02, use_connection_pooling=True):This is misleading since it gives the impression that API calls are being retried by default, and that these retries are configurable. However, they are not, since all calls in the client use HTTP POST methods, and the What's even more confusing is that the log message does print the (fake) number of retries: You can validate this by setting a large Judging from the presence of the parameters in the API client I assume the authors' intentions were to retry |
|
If you'd like to work around this while waiting for an upstream fix, try if this works: client = APIClient(
key=api_key,
region=Regions.EU,
retries=5,
backoff_factor=1.0,
)
def build_session_with_retries(self):
session = super(APIClient, self)._build_session()
session.headers["Authorization"] = "Bearer {key}".format(key=self.key)
# Retry request a number of times before raising an exception
# also define backoff_factor to delay each retry. Retry even on
# non-idempotent methods.
session.mount(
"https://",
HTTPAdapter(
max_retries=Retry(
total=self.retries,
backoff_factor=self.backoff_factor,
allowed_methods=None,
)
),
)
return session
client._build_session = types.MethodType(
build_session_with_retries, self.client
)It replaces the original |
|
Yes we came to the same conclusion about retrying POST requests. That cleared up the issue for us. Unfortunately, emails are non-idempotent by nature, so retrying opens up the possibility of duplicate sends. CIO support has not been very helpful here unfortunately, neither investigating the connection reset issue nor addressing the fact that retrying an API request that sends an email, without any kind of protection from duplicate sends, is a bad idea. That said we've never had a customer complaint about duplicate emails, so could be it's just a theoretical concern. |
|
I just came across the following notes in the README:
I conclude that they are aware of the issues, and have a feeling they will not merge my PR in that case. If the issue exists due to a combination of connection pooling and lack of retries, and disabling connection pooling is indeed an alternative, then I would propose CustomerIO devs start to take out the parameters from the I wouldn't be surprised if this is due to a mismatch in urllib's default client side connection (pool ) timeout and CustomerIO's server side connection timeout, disconnecting earlier. That explains why we're seeing it only on low volume apps that keep running, and never on e.g. batch jobs. |
|
For anyone else experiencing this issue: We were experiencing this issue for a while, and by taking advantage of the parameter (released in version 1.6) @skion references in the comment above and initializing with |
Hi All,
Been using customerio-python for a fair bit in production environment. From time-time-time we would get the following error:
Cannot send email for xxxxx message id xx ** Failed to receive valid response after 3 retries.**
Check system status at http://status.customer.io.
The same payload/user would go through at another instance.
Python 3.8
Has anyone encountered this before?
Would a simple remedy be adding a retry wrapper on top?
The text was updated successfully, but these errors were encountered: