Skip to content

Latest commit

 

History

History
287 lines (176 loc) · 7.73 KB

portroach.pod

File metadata and controls

287 lines (176 loc) · 7.73 KB

NAME

portroach - Tool to scan for new versions of OpenBSD ports.

SYNOPSIS

portroach build

while (lazy) {
    portroach rebuild
    portroach check
    portroach showupdates
}

DESCRIPTION

Portroach is an automated system designed to search for new versions of software available in the OpenBSD ports tree. It is primarily designed for use by OpenBSD port maintainers, who can avoid trailing around dozens of websites looking for updates. However, I hope that others might find it useful too.

SYSTEM REQUIREMENTS

The following software is required to run Portroach:

- PostgreSQL or SQLite
- POSIX-compatible system
- The OpenBSD ports tree
- Berkeley make
- Perl 5.10+

Plus we need a few Perl modules:

- URI
- DBD::Pg or DBD::SQLite
- Net::FTP
- Proc::Queue
- LWP::UserAgent
- MIME::Lite
- XML::Feed
- Try::Tiny
- JSON

SQLite support is currently limited to non-forking mode only. That is, if you decide to use SQLite, Portroach will only check one port at a time; this will severely limit Portroach's speed/efficiency. It is therefore suggested that SQLite only be used for relatively light workloads.

INITIAL SET-UP

Initialise Database

The recommended database backend is PostgreSQL.

Option One: PostgreSQL

Create database:

# createuser -U postgres -P portroach
# createdb -U postgres portroach

Execute the included pgsql_init.sql script via psql:

# psql portroach portroach < sql/pgsql_init.sql

This will create the database tables for you.

Option Two: SQLite

Create a database file with the included script:

# sqlite3 /var/db/portroach.db < sql/sqlite_init.sql

Configure Portroach

Review portroach.conf, and check it suits your needs. The defaults should be reasonable for most people. You can reduce num_children and workqueue_size if you don't want Portroach sucking up all your resources.

Please note that Portroach's internal defaults differ from the defaults in portroach.conf - this is because without a config file, Portroach tries to be "portable" and use its own directory for storing things under, whereas if a config file is found, it assumes it is installed and being used "system-wide".

Any of the options in portroach.conf can also be set on the fly on the command line. For example:

$ portroach --precious_data --num_children=8

Update Ports Tree

Ensure your ports tree is up to date.

Populate Database

We need now to populate the database with the software we want to check.

Option One: OpenBSD

To start off, run:

$ portroach build

This takes around 70 minutes for me. Basically, Portroach is extracting all the information it needs from the ports tree. Ports (by virtue of make) is slow; the database we're building is much faster. After this initial build, we will do incremental 'builds', only updating what has changed.

REGULAR OPERATION

Update Ports Tree

Ensure your ports tree is up to date.

Incremental Database Update

If sqlports was updated since your last build/rebuild, ensure Portroach knows about the changes:

$ portroach rebuild

Run Version Checks

$ portroach check

This will instruct Portroach to search for new distfiles for each port in the database. It will take several hours for a complete ports tree scan.

View Results

Now you've got some results, you can view them.

Option One: HTML Reports

$ portroach generate

This will put HTML pages inside html_data_dir - existing pages will be deleted.

Option Two: E-Mail Reports

$ portroach mail

This will send out an e-mail message to the maintainers of ports with updates. The e-mail messages are opt-in; you will need to add addresses to the database before any e-mails are sent out.

Option Three: Console Summary

$ portroach showupdates

This will output a summary of software with outstanding updates. It is recommended if you're checking a limited set of software/ports.

ADDING E-MAIL ADDRESSES

If you want to send e-mail reports to maintainers of updated of software, the e-mail addresses need to be registered with Portroach. This is a safeguard to ensure no one gets e-mails they don't want.

Use the following to manage these e-mail "subscriptions":

$ portroach add-mail [email protected]

$ portroach remove-mail john@localhost

$ portroach show-mail

CLEANING

Ports that have been removed are not automatically removed from the database, so over time the database may contain data that doesn't refer to anything anymore. In order to cleanup the database the "prune" command can be used:

$ portroach prune

Currently this will ensure the following:

- that all ports in the database still exist in the ports tree

UPGRADING

When upgrading, check the sql directory for any relevant database schema upgrade scripts. If there were multiple schema updates between the previous version of Portroach and the version to which you have upgraded, be sure to run each script in sequence to arrive at the latest database version.

CHECKING ALGORITHM

For anyone interested in how Portroach operates, here is a high-level summary of the checking algorithm in use:

Test 1:
  1) Order master sites using previous reliability data.
  2) Attempt to get an FTP listing or web server index from each site.
  3) Extract version from files found; compare to current version.
  4) Skip other tests if new or current version is found.

Test 2:
  1) Increment each part of the port's version string and attempt to
     download file, e.g. for 1.4.2, try 2.0.0, 1.5.0 and 1.4.3

The last test is not yet included in Portroach, but I may add it at some point, depending on the results of testing:

Test 3:
  1) Locate port's WWW page and spider site in an attempt to find a
     page that looks like a "download area". Scan page for possible
     links to new files.

BUGS AND LIMITATIONS

  • Portroach tries to make a reasonable guess when it encounters version strings in a different format to the original distname (e.g. 3.2, 3.6-pre7), but this is difficult and error-prone since vendor version schemes vary wildly.

    The only real problem at the moment is version strings which seem to to count backwards (e.g. 2.11 -> 2.2).

  • There's some difficulty in deciding what to do with trailing zeros in version guesses. Currently, they are left intact, but this is not always going to be the right course of action. In other words, from 4.3.9, will the next major version be 4.4.0 or 4.4?

  • The restrict_* variables don't affect generate/mail.

  • Portroach doesn't handle ports with multiple distfiles very well.

  • At least one port (archivers/zip, as of 2010-04-28) doesn't provide a version string in the vendor's format. portroach doesn't know what to do in this case, although the version could theoretically be ascertained from the distfile name.

HISTORY

Portscout was originally written by Shaun Amott for the FreeBSD ports tree. In 2014 Jasper Lievisse Adriaanse added support for OpenBSD. It was then decided to fork and rename the project to Portroach to indicate a new direction, focusing on OpenBSD allows Portroach to integrate better without carrying a burden of backwards compatibility with the FreeBSD tree.

CREDITS

Many thanks to Shaun Amott for Portscout from which portroach is derived. Portroach wouldn't exist if it wasn't for Shaun's great work on portscout.

Thanks for Antoine Jacoutot for suggesting the new project name, Portroach.

COPYRIGHT

All rights reserved.