Updated scraper for URL changes. #4
Conversation
There was a problem hiding this comment.
Thanks for working on this! It's not all clear to me from the commit messages what you're changing and why. "URL changes" could mean anything for example and you're rewriting the code in a not-completely-obvious way while adding functionality which makes reviewing harder. It's great when a commit does an atomic change that is self contained. Personally I also separate refactoring into their own commits so you anyone can easily verify those and also more easily eyeball the real changes that are happening elsewhere
scraper.rb
Outdated
| @@ -35,5 +34,5 @@ | |||
|
|
|||
| # puts record.inspect | |||
| ScraperWiki.save_sqlite(['council_reference'], record) | |||
| puts "Saving " + council_ref | |||
| puts "Saving " + council_ref.strip() | |||
There was a problem hiding this comment.
If council_ref needs whitespace remove why don't you do that before the data is saved to the database?
| @@ -1,38 +1,67 @@ | |||
| require 'scraperwiki' | |||
| require 'mechanize' | |||
|
|
|||
| starting_url = 'http://www.sorell.tas.gov.au/publications/currently-advertised-applications/' | |||
| def scrape() | |||
There was a problem hiding this comment.
You're mixing up refactoring with some changes which are not at all clear from the commit message. So, this whole commit is difficult for me to review
| @@ -6,6 +6,13 @@ def scrape() | |||
|
|
|||
| agent = Mechanize.new | |||
|
|
|||
| if ENV["MORPH_AUSTRALIAN_PROXY"] | |||
There was a problem hiding this comment.
This change (or something similar) is already being introduced by another PR #3
Scraper now captures address and council reference correctly.
Also added support of proxy use.
Disabled SSL check as the previous run of this scraper encountered an SSL check error.