-
Notifications
You must be signed in to change notification settings - Fork 17
/
CHANGELOG
53 lines (35 loc) · 1.55 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
#############################
crawler4py
ENTRY 5
* Fixed a bug in Frontier Add on resume to check if the urls are valid again.
--Rohan Achar - Feb 4th, 2016 - 9:15PM
#############################
crawler4py
ENTRY 4
* Fixed a bug in AddToFrontier where duplicate urls at different depths were being accepted into the Frontier. Bug found by Rimoun Ghaly
* Tested on py2.6, py2.7, py3.5
--Rohan Achar - Jan 25t, 2016 - 12:08AM
#############################
crawler4py
ENTRY 3
* Made project into a package and Config an abstract class.
* Changed from deprecated sets.Set() to built-in set()
* Made changes to make it cross compatible with both python 2 and python 3
* Changed frontier to an ordered set and allowed for depth first or breadth first traversals.
-- Rohan Achar - May 11, 2014 - 9:00AM
#############################
crawler4py
ENTRY 2
* Changed the Seed from attribute to method so that more preparation work can be done.
* Allowed Ignore Robot rules config entry.
* Url encoded to utf-8 to avoid unicodeerrors.
* Support for Basic Authentication.
-- Rohan Achar - May 9, 2014 - 10:15PM
#############################
crawler4py
ENTRY 1
* Shifted the extraction of next links to config. This supports greater flexibility in type of data. Case example: The parser can be used to fetch data from API iteratively by getting the next link correctly from the data previously got.
* Added timestamp messages
* Made it less python 2 specific. Syntax should work for python 3 as well. Not tested.
-- Rohan Achar - May 9, 2014 - 10:15AM
#############################