portroach - Tool to scan for new versions of OpenBSD ports.
portroach build
while (lazy) {
portroach rebuild
portroach check
portroach showupdates
}
Portroach is an automated system designed to search for new versions of software available in the OpenBSD ports tree. It is primarily designed for use by OpenBSD port maintainers, who can avoid trailing around dozens of websites looking for updates. However, I hope that others might find it useful too.
The following software is required to run Portroach:
- PostgreSQL or SQLite
- POSIX-compatible system
- The OpenBSD ports tree
- Berkeley make
- Perl 5.10+
Plus we need a few Perl modules:
- URI
- DBD::Pg or DBD::SQLite
- Net::FTP
- Proc::Queue
- LWP::UserAgent
- MIME::Lite
- XML::Feed
- Try::Tiny
- JSON
SQLite support is currently limited to non-forking mode only. That is, if you decide to use SQLite, Portroach will only check one port at a time; this will severely limit Portroach's speed/efficiency. It is therefore suggested that SQLite only be used for relatively light workloads.
The recommended database backend is PostgreSQL.
Create database:
# createuser -U postgres -P portroach
# createdb -U postgres portroach
Execute the included pgsql_init.sql script via psql
:
# psql portroach portroach < sql/pgsql_init.sql
This will create the database tables for you.
Create a database file with the included script:
# sqlite3 /var/db/portroach.db < sql/sqlite_init.sql
Review portroach.conf, and check it suits your needs. The defaults should be reasonable for most people. You can reduce num_children
and workqueue_size
if you don't want Portroach sucking up all your resources.
Please note that Portroach's internal defaults differ from the defaults in portroach.conf - this is because without a config file, Portroach tries to be "portable" and use its own directory for storing things under, whereas if a config file is found, it assumes it is installed and being used "system-wide".
Any of the options in portroach.conf can also be set on the fly on the command line. For example:
$ portroach --precious_data --num_children=8
Ensure your ports tree is up to date.
We need now to populate the database with the software we want to check.
To start off, run:
$ portroach build
This takes around 70 minutes for me. Basically, Portroach is extracting all the information it needs from the ports tree. Ports (by virtue of make) is slow; the database we're building is much faster. After this initial build, we will do incremental 'builds', only updating what has changed.
Ensure your ports tree is up to date.
If sqlports was updated since your last build/rebuild, ensure Portroach knows about the changes:
$ portroach rebuild
$ portroach check
This will instruct Portroach to search for new distfiles for each port in the database. It will take several hours for a complete ports tree scan.
Now you've got some results, you can view them.
$ portroach generate
This will put HTML pages inside html_data_dir
- existing pages will be deleted.
$ portroach mail
This will send out an e-mail message to the maintainers of ports with updates. The e-mail messages are opt-in; you will need to add addresses to the database before any e-mails are sent out.
$ portroach showupdates
This will output a summary of software with outstanding updates. It is recommended if you're checking a limited set of software/ports.
If you want to send e-mail reports to maintainers of updated of software, the e-mail addresses need to be registered with Portroach. This is a safeguard to ensure no one gets e-mails they don't want.
Use the following to manage these e-mail "subscriptions":
$ portroach add-mail [email protected]
$ portroach remove-mail john@localhost
$ portroach show-mail
Ports that have been removed are not automatically removed from the database, so over time the database may contain data that doesn't refer to anything anymore. In order to cleanup the database the "prune" command can be used:
$ portroach prune
Currently this will ensure the following:
- that all ports in the database still exist in the ports tree
When upgrading, check the sql directory for any relevant database schema upgrade scripts. If there were multiple schema updates between the previous version of Portroach and the version to which you have upgraded, be sure to run each script in sequence to arrive at the latest database version.
For anyone interested in how Portroach operates, here is a high-level summary of the checking algorithm in use:
Test 1:
1) Order master sites using previous reliability data.
2) Attempt to get an FTP listing or web server index from each site.
3) Extract version from files found; compare to current version.
4) Skip other tests if new or current version is found.
Test 2:
1) Increment each part of the port's version string and attempt to
download file, e.g. for 1.4.2, try 2.0.0, 1.5.0 and 1.4.3
The last test is not yet included in Portroach, but I may add it at some point, depending on the results of testing:
Test 3:
1) Locate port's WWW page and spider site in an attempt to find a
page that looks like a "download area". Scan page for possible
links to new files.
Portroach tries to make a reasonable guess when it encounters version strings in a different format to the original distname (e.g. 3.2, 3.6-pre7), but this is difficult and error-prone since vendor version schemes vary wildly.
The only real problem at the moment is version strings which seem to to count backwards (e.g. 2.11 -> 2.2).
There's some difficulty in deciding what to do with trailing zeros in version guesses. Currently, they are left intact, but this is not always going to be the right course of action. In other words, from 4.3.9, will the next major version be 4.4.0 or 4.4?
The restrict_* variables don't affect generate/mail.
Portroach doesn't handle ports with multiple distfiles very well.
At least one port (archivers/zip, as of 2010-04-28) doesn't provide a version string in the vendor's format. portroach doesn't know what to do in this case, although the version could theoretically be ascertained from the distfile name.
Portscout was originally written by Shaun Amott for the FreeBSD ports tree. In 2014 Jasper Lievisse Adriaanse added support for OpenBSD. It was then decided to fork and rename the project to Portroach to indicate a new direction, focusing on OpenBSD allows Portroach to integrate better without carrying a burden of backwards compatibility with the FreeBSD tree.
Many thanks to Shaun Amott for Portscout from which portroach is derived. Portroach wouldn't exist if it wasn't for Shaun's great work on portscout.
Thanks for Antoine Jacoutot for suggesting the new project name, Portroach.
- Copyright (C) 2005-2011, Shaun Amott <[email protected]>.
- Copyright (C) 2014-2015 Jasper Lievisse Adriaanse <[email protected]>.
All rights reserved.