Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental back-off to circumbent Github abuse detection #4

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

timfel
Copy link

@timfel timfel commented Nov 20, 2015

When migrating larger projects, Github's abuse detection triggers at some point. To still allow the migration to go on, incrementally back off, and also check the rate limits for the API to check how long we'll have to wait.

@gunchleoc
Copy link

I tried to use this patch (without using the new script, since I already had the JSON data) to migrate > 3000 issues and it failed before finishing second pass for all issues.

second pass on issue Ingame README needs review
Traceback (most recent call last):
  File "./bin/lp2gh-import-bugs", line 30, in <module>
    o = bugs.import_(repo, data, milestones_map=milestones_map)
  File "<path>/lp2gh/lp2gh/bugs.py", line 267, in import_
    limit_retry(e, repo, lambda: comments.append(body='%s\n%s' % (by_line, msg['content'])))
  File "<path>/lp2gh/lp2gh/bugs.py", line 49, in limit_retry
    return try_block()
  File "<path>/lp2gh/lp2gh/bugs.py", line 267, in <lambda>
    limit_retry(e, repo, lambda: comments.append(body='%s\n%s' % (by_line, msg['content'])))
  File "<path>/lp2gh/github3/client.py", line 70, in append
    rv = self.client.post(self.url, **kw)
  File "<path>/lp2gh/github3/request.py", line 41, in post
    return self._opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 402, in _read_status
    raise BadStatusLine("No status line received - the server has closed the connection")
httplib.BadStatusLine: No status line received - the server has closed the connection

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants