It gives RecursionError: maximum recursion depth exceeded Error

[vorarush]

For sites with many links it gives "RecursionError: maximum recursion depth exceeded" error. I think it uses tail recursion and that is why it stucks at default recursion limit of python. The logic needs to be improvised to not face this limit issue. Can you help?

Traceback (most recent call last):
File "email_crawler.py", line 146, in
crawl.crawl()
File "email_crawler.py", line 40, in crawl
self.crawl()
File "email_crawler.py", line 40, in crawl
self.crawl()
File "email_crawler.py", line 40, in crawl
self.crawl()
[Previous line repeated 958 more times]
File "email_crawler.py", line 36, in crawl
self.parse_url(url)
File "email_crawler.py", line 62, in parse_url
response = requests.get(current_url, headers=self.headers)
File "\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "\lib\site-packages\requests\sessions.py", line 665, in send
history = [resp for resp in gen]
File "\lib\site-packages\requests\sessions.py", line 665, in
history = [resp for resp in gen]
File "\lib\site-packages\requests\sessions.py", line 245, in resolve_redirects
**adapter_kwargs
File "\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "\lib\site-packages\requests\adapters.py", line 533, in send
return self.build_response(request, resp)
File "\lib\site-packages\requests\adapters.py", line 265, in build_response
response = Response()
File "\lib\site-packages\requests\models.py", line 608, in init
self.headers = CaseInsensitiveDict()
File "\lib\site-packages\requests\structures.py", line 46, in init
self.update(data, **kwargs)
File "\lib_collections_abc.py", line 839, in update
if isinstance(other, Mapping):
File "\lib\abc.py", line 182, in instancecheck
if subclass in cls._abc_cache:
RecursionError: maximum recursion depth exceeded

[osean-man]

I have encountered this same issue.