ConnectionResetError: [Errno 104] Connection reset by peer

[sunilrk077]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Hello Everyone,

Suddenly getting below error while invoking API. This was working fine until 07/20, Saturday. Anyone facing same error?

Note: We are encountering this issue on all the AWX instances throughout the all locations.

AWX INFO *****************
cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:

Find the latest tag here: https://github.com/ansible/awx-operator/releases

• github.com/ansible/awx-operator/config/default?ref=0.29.0

• awx-pro.yaml

• my-ingress.yml

Set the image tags to match the git version from above

images:

• name: quay.io/ansible/awx-operator
  newTag: 0.29.0

Specify a custom namespace in which to install AWX

namespace: awx-lvn

Error Info ************************

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in validate_conn
conn.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

AWX Operator version

0.29.0

AWX version

21.6.0

Kubernetes platform

minikube

Kubernetes/Platform version

Minikube: 1.27

Modifications

no

Steps to reproduce

job_launch_url="$AWX_HOST/api/v2/job_templates/$ext_tempt_id/launch/"
job_id=$(curl -k -H "Authorization: Bearer $AWX_TOKEN" -X POST "$job_launch_url")
ext_job_id=$(extract_value 'id' "$job_id")
echo "Job '$ext_job_id' launched from template '$TEMPLATE_NAME'."

Issue occurring after job is launched through API from the client, the job launches which hits AWX server but then connection is getting rejected when AWX is trying to connect client

Error:

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in validate_conn
conn.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

Expected results

AWX should launch job on the target client.

Actual results

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in validate_conn
conn.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

Additional information

No response

Operator Logs

No response

[djyasin]

Hello,
Have you checked to see if your connection was reset at any time? That could possibly cause this to occur. Please let us know if this resolves the issue for you!

[sunilrk077]

Thanks for response. connection is stable, never reseted. but it fails everytime when we invoke the API.