PY3: Twisted Dependencies

[PORTED] twisted.internet.base

There is an open issue: https://twistedmatrix.com/trac/ticket/7540

ThreadedResolver

• scrapy/resolver.py

[PORTED] twisted.internet.defer

• scrapy/contrib/feedexport.py
• scrapy/contrib/downloadermiddleware/retry.py
• scrapy/contrib/pipeline/files.py
• scrapy/core/downloader/__init__.py
• scrapy/core/downloader/handlers/__init__.py
• scrapy/core/downloader/handlers/http11.py
• scrapy/core/downloader/webclient.py
• scrapy/core/engine.py
• scrapy/core/scraper.py
• scrapy/crawler.py
• scrapy/mail.py
• scrapy/resolver.py
• scrapy/shell.py
• scrapy/utils/decorator.py
• scrapy/utils/defer.py
• scrapy/utils/serialize.py
• scrapy/utils/testproc.py
• scrapy/contrib/pipeline/media.py
• scrapy/utils/request.py
• scrapy/utils/signal.py

[PORTED] twisted.internet.error

• scrapy/utils/reactor.py
• scrapy/core/downloader/handlers/http11.py

[PORTED] twisted.internet.reactor / twisted.internet.default

They are basically a wrapper around twisted.internet.epollreactor and twisted.internet.selectreactor. twisted/internet/posixbase.py is (mostly?) ported.

• scrapy/contrib/closespider.py
• scrapy/core/downloader/__init__.py
• scrapy/core/downloader/handlers/ftp.py
• scrapy/core/downloader/handlers/http10.py
• scrapy/core/downloader/handlers/http11.py
• scrapy/crawler.py
• scrapy/mail.py
• scrapy/shell.py
• scrapy/utils/benchserver.py
• scrapy/utils/defer.py
• scrapy/utils/ossignal.py
• scrapy/utils/reactor.py
• scrapy/utils/testproc.py
• scrapy/utils/testsite.py

[PORTED] twisted.internet.threads

• scrapy/contrib/feedexport.py
• scrapy/contrib/pipeline/files.py
• scrapy/shell.py
• scrapy/utils/decorator.py

[PORTED] twisted.internet.task

• scrapy/contrib/logstats.py
• scrapy/contrib/memusage.py
• scrapy/core/downloader/__init__.py
• scrapy/utils/defer.py

[PORTED] twisted.python.failure

Failure

• scrapy/command.py
• scrapy/contrib/pipeline/media.py
• scrapy/core/engine.py
• scrapy/core/scraper.py
• scrapy/core/spidermw.py
• scrapy/logformatter.py
• scrapy/utils/defer.py
• scrapy/utils/signal.py

[PORTED] twisted.internet.ssl

ClientContextFactory

This is going to be changed to use a class that implements IPolicyForHTTPS.

• scrapy/core/downloader/contextfactory.py
• scrapy/mail.py

twisted.web

[Ported] http_headers

• scrapy/core/downloader/handlers/http11.py

[Ported] iweb

Interfaces use zope.interface which is already ported to Python 3.

• scrapy/core/downloader/handlers/http11.py

[Ported] http

• scrapy/core/downloader/handlers/http11.py
• scrapy/core/downloader/webclient.py
• scrapy/utils/response.py

[Partly Ported] client

There are two open issues:

1. https://twistedmatrix.com/trac/ticket/7407
2. https://twistedmatrix.com/trac/ticket/6197

• scrapy/core/downloader/webclient.py

[Partly Ported] server

• scrapy/utils/benchserver.py
• scrapy/utils/testsite.py

[Ported] resource

• scrapy/utils/benchserver.py
• scrapy/utils/testsite.py

[Ported] static

• scrapy/utils/testsite.py

[Ported] util

• scrapy/utils/testsite.py

twisted.conch

Needs porting. Another alternative solution would be moving the telnet module into a contrib module. It's useful for debugging, but it's not a core part of Scrapy.

• scrapy/telnet.py

twisted.protocols.ftp

Needs porting. It's a huge module, it may take a while to port it right. FTP servers are less popular these days, perhaps we could defer porting it.

• scrapy/core/downloader/handlers/ftp.py

twisted.mail.smtp

Needs porting.

• scrapy/mail.py

[PORTED] twisted.internet.protocol

Most of them interfaces and base classes.

• scrapy/telnet.py
• scrapy/utils/testproc.py
• scrapy/core/downloader/handlers/http11.py
• scrapy/core/downloader/handlers/ftp.py