forked from kop1/newznab
-
Notifications
You must be signed in to change notification settings - Fork 2
/
README.txt
210 lines (180 loc) · 11.6 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
ABOUT
Newznab is a PHP/Smarty application, which supports the indexing of usenet headers
into a mysql database and provides a simple web based search interface onto the data.
It includes simple CMS facilities, SEO friendly URLs and is designed with the intention
of allowing users to create a community around their index.
For information on how to install, please refer to INSTALL.txt
To discuss use irc.synirc.net #newznab
Newznab is licensed under terms of the GNU General Public License. For details, please
refer to LICENSE.txt.
HOW IT WORKS
usenet groups are specified, message headers (binaries and parts) are downloaded for the
groups, releases are created from completed sets of binaries by applying regex to the message subject.
releases are categorised by regexing the message subject. metadata from tvrage, tmdb, imdb and amazon
are applied toeach created release. after a configurable number of days the header data is deleted
from the database, but the releases remain.
CHOOSING NEWSGROUPS
groups can be manually entered if you know the name. groups can also be bulk added when
specified as a regular expression. for example if you want to index the groups alt.bin.blah.*
and alt.bin.other use the value 'alt.bin.blah.*|alt.bin.other'.
UPDATING INDEX (populating binaries + parts)
the recommended way to schedule updates is via the dos and unix start scripts in
/path/to/newznab/misc/update_scripts/. make sure you set the paths correctly to your installation.
CATEGORISATION
most categorisation of releases is done at the time of applying the regex. however if no category
is supplied for a regex then \www\lib\category.php contains the logic which attempts to map a
release to a site category. site categories are used to make browsing nzbs easier. add new categories
by updating the category table, and adding a new Category::constant. Then map it in the
function determineCategory()
MISSING PARTS
when headers are requested from the usenet provider, they are asked for in number ranges
e.g. 1-1000, 1001-2000 etc. for various reasons sometimes the provider does not return
a header, this is not always because the header does not exist, there may be some synchronisation
going on at the providers end. if a header is requested but not returned, we store a record of this
in the table partrepair. each time update_binaries is ran an attempt is made to go back and get the
missing parts. if after five attempts the parts can still not be obtained, newznab gives up.
when update_releases runs, if a release is seen to have missing parts it will not be released until
four hours after it was uploaded to usenet. this is so a chance has been made to repair all missing
parts. after four hours a release will be created anyway and its down to the quality of the par files
to determine whether a release can be correctly unpacked.
BACKFILLING GROUPS
since most usenet providers have 800+ days of retention indexing all that information in one shot
is not practical. newznab provides a backfill feature that allow you to index past articles once
your initial index has been built. to use the feature first set the back fill days setting in the group(s)
to be backfilled to the number of day you wish to go back, making sure to set it higher than the number
of days listed in the first post column. once set run the backfill.php script in misc/update_scripts.
groups can be backfilled to a particular date using the script misc/update_scripts/backfill_date.php
using the syntax php backfill_date.php 2011-05-15 alt.binaries.groupname
REGEX MATCHING
releases are created by applying regexs to binary message subjects. different regexes
are applied to binaries from different newsgroups. catchall regexes are applied to any
binaries left unmatched after the group specific matching. a category can be associated
with a regex, which will allow the processing of groups like inner-sanctum which contain a
combination of different binary types.
REGEX UPDATING
regexes in the system in the range 0-10000 are system defined and are updated centrally.
everytime processreleases is ran, a check will be performed to see if you have the latest regexs.
if you do not want this check to be made then set site.latestregexurl to null
NZB FILE STORAGE
nzbs are saved to disk gzipped at the location specified by site.nzbpath in subdirs based on the
first char of the release guid, this just makes the dirs a bit easier to manage when you have thousands
of nzb.gz files. the default path is /website/../nzbfiles
SSL USENET CONNECTION
Install the OpenSSL extension, set config.php define ('NNTP_SSLENABLED', true);
IMPORTING/EXPORTING NZBS
.nzb files can be imported from the admin interface (or cli). importing is a convenient way to fill the
index without trawling a large backdated number of usenet messages. after running an import
the processReleases() function must be run to create valid releases. nzbs can also be exported
based on system categories.
GOOGLE ADS/ANALYTICS
to integrate google analytics and adsense provide enter the adsense ad module ids
into the site table for the sidepanel (right), searchbox (bottom of menu).
providing an analytics id will include the analytics js in the footer
ADMIN
admin functions all live in /admin/ which is only accessible by users with admin role.
set users.role to be 2 on the users you wish to be admins
TVRAGE
after releases.processReleases() is called, an attempt is made to determine the tvrageids for every
release which looks like its tv. this also works out the series/episode columns. the data in the
tvrage table will become populated from best guesses from the rage search api. if some of these
guesses are wrong, you can manually edit the rage data in the admin interface, and use the
remove link to wipe any releases which have that rageid and then manually call 'process tv' which
will attempt to relink rage data. when a new release is created it goes in with a release.rageid of -1
when tv is processed, the rageid either goes to the best guess, or to -2, which indicates no match
could be made, or the release isnt percieved to be tv.
TVRAGE - SERIES/EPISODE INFO
information about a particular episode of a series is requested from services.tvrage.com a shared
newznab api key is used to retrieve this data. it assigns an 'aired date' to each release if a match
is found.
NFO
nfos are attempted to be retrieved using a queuing method. there will be a number of attempts to get
an nfo before giving up
IMDB/TMDB
if enabled, and if an imdb id is found in the nfo, the application will attempt to use that imdb id to
get general data about the movie (title, year, genre, covers etc) from themoviedb.org. If no entry is
available from tmdb then an attempt to gather the info from imdb.com is made. any results are stored
in the moveinfo table, with covers/backdrops being saved to the images/covers/.
3RD PARTY API KEYS
in order to do lookups to tmdb and amazon, api keys are used. newznab is shipped with some default
keys, but due to the restrictions on use of api's, it is strongly suggested you go and get your own
api keys and save them in the site edit page.
CONTENT/CMS
pages can be added to the site with seo friendly urls via the /admin/ edit content links
SKINNING/THEMES
avoid custom edits to code and stylesheets to make updating painless.
override any styles by creating a folder \www\theme\<yourtheme>\style.css. stick any custom images in
\www\theme\<yourtheme>\images\
pick the theme in the admin\site-edit page. your styles should override the existing style sheet, which
should always be loaded in.
API
www.sitename.com/api? provides api access to query and retrieve nzbs.
call www.sitename.com/apihelp to see help doc with all available options. users either have to be
logged in or provide their rsstoken. users can use their rsstoken to access both rss + api.
full details of the api and how to implement it are provided in /misc/api/
FAQS
* authorisation rejected from nntp server - check you have enough available connections
not currently in use
* white screen - probably php error not being displayed to browser or session timed out and 403 being throw
* Lots of binary headers processed but few releases created - The binary headers downloaded do not match
the regexes used to create a release. The message headers must follow popular formats in order for releases
to be created.
* search and rawsearch requests lose page css styling - use the provided apache vhost settings.
* Server did not return article numbers 1234567 - this isnt necessarily a bad thing, see section on missing parts
* connection timed out. Reconnecting... Cannot connect to server *******: Already connected, disconnect first! -
disable compressed headers
* session error during install step1 - set register_globals to off.
* Warning: Wrong parameter count for strstr() in newznab\www\lib\TMDb.php on line 354 - wrong php version, requires 5.3+
* Strict Standards: Non-static method PEAR::isError() should not be called statically - disable strict error reporting
* Error 502 Bad Gateway - error at $cfg->pearCheck = @include('System.php'); solved by adding in open_basedir path to pear
* Call to undefined function curl_init() - Make sure you are using the right php.ini file. If you are using WAMP, then
the php.ini file that apache uses is in the apache /bin folder (not the php.ini in wamp/php). The php cli will use
the first php.ini it can find in the windows path environment variable. In my case, this was an old version in
another php directory I set up. Once I deleted that, it used the version in the /wamp/php directory.
DEBUGGING
switch php.ini error_reporting to E_ALL and ensure logging to browser is enabled.
DEVELOPMENT
\db\schema.sql is latest database schema. should be able to rerun in and create new blank schema
\misc\ for general docs and useful info, nothing in here is referenced by the application
\misc\update_scripts\ shell and batch scripts and php files to call the updating of index from cli
\nzbfiles default folder for all gzipped nzbs to be stored
\www\install installer files
\www\lib\framework few general classes for db/http code
\www\lib\smarty copy of a fairly recent smarty lib
\www\lib\ all classes used in the app, typically named same as its database entity
\www\covers\ all covers downloaded for releases
\www\views\templates\admin all templates used by the admin pages
\www\views\templates\frontend all templates used by the user pages
\www\pages\ controllers for every frontend page in the system
\www\admin\ all php pages used by the admin
\www\theme\<yourtheme> blank area for implementation specific ui customisations
\www\views\scripts\ js dumping ground
HALL OF FAME
(just some of the) people who've helped along the way.
[email protected] regexs,sessions
[email protected] regexs
fatfecker@newznab mediainfo ffmpeg tv
[email protected] password,hash
[email protected] yenc,nntp,bokko,dev
[email protected] anidb
bb@newznab dev
keyvan@newznab backfill
[email protected] testing,icons
midgetspy@sickbeard rage integration,api
ueland@newznab installer
hecks@tvnzb rar api
michael@newznab dev
sakarias@newznab testing
pairdime@sabnzbd jquery,css
pmow@sabnzbd headers,backfill
bigdave@newznab testing
duz@sabnzbd yenc
inpheaux@sabnzbd design,nzb
spooge@newznab testing
sy@newznab testing, regexs, amazon
magegminds@newznab lighttpd rewrite rules
trizz@newznab lighttpd rewrite rules
fubaarr@newznab testing
mobiKalw@newznab testing
crudehung@newznab nginx rewrite rules
www.famfamfam.com icons