smarc.7

.\" smarc.7 was written by Omar Polo <op@openbsd.org> and is placed in
.\" the public domain.  The author hereby disclaims copyright to this
.\" source code.
.Dd September 4, 2023
.Dt SMARC 7
.Os
.Sh NAME
.Nm smarc
.Nd mailing list web archive generation system
.Sh DESCRIPTION
.Nm
is a system to incrementally generate a web archive for a mailing list
and optionally provide search capabilities.
The generated archive is a set of HTML files that can be served as-is,
while searching requires the use of a FastCGI application.
At a higher level,
.Nm
is made of three components:
.Pp
.Bl -tag -width msearchd_8_ -compact -offset indent
.It Xr smarc 1
to generate the static HTML files from a maildir.
.It Xr smingest 1
to import emails into a sqlite3 database.
.It Xr msearchd 8
to provide search results.
.El
.Sh INITIAL SETUP
There are several step necessary to initialize the web archive:
.Pp
.Bl -enum -compact -offset indent
.It
Create and populate the output directory.
.It
Customize the templates.
.It
Prepare the maildir.
.It
Generate the web archive.
.It
Set up the database for searching.
.It
Configure the web server.
.El
.Pp
It is recommended to use a dedicate user.
Commands to be run as a unpriviledged user are preceded by a dollar sign
.Sq $ ,
while commands requiring superuser privileges by a hash mark
.Sq # .
Hereafter, it will be assumed that the local user is called
.Sq smarc .
.Ss 1. Create and populate the output directory
The web archive is made of several static files, mostly HTML, that needs
to be served by a web server like
.Xr httpd 8 .
.Pa /var/www/smarc
is the default location, but a different path can be used.
To prepare it, issue:
.Bd -literal -offset indent
# mkdir -p /var/www/smarc
# chown smarc /var/www/smarc
.Ed
.Pp
Then copy the CSS file, optionally tweaking it.
.Pp
.Dl $ cp /usr/local/share/examples/smarc/style.css /var/www/smarc
.Pp
Copy also other eventual assets
.Pq e.g.\& logo images
here as well.
.Ss 2. Customize the templates
The default templates are installed at
.Pa /etc/smarc .
Since these are anonymous, they need to be tweaked to include
information about the mailing list.
.Pp
Care should be taken when editing these files after generating the
archive since existing pages won't be automatically updated.
The cachedir
.Pq see Xr smarc 1
needs to be deleted and the web archive generated again.
.Xr msearchd 8
has to be stopped and restarted as well.
.Ss 3. Prepare the maildir
The maildir with the mailing list entries needs to be populated.
It is assumed to be at
.Pa ~/Mail/smarc
by default, but a different path can be used.
.Ss 4. Generate the web archive
.Xr smarc 1
can be finally used to generate the web archive.
The first run may take a while, depending on the size of the maildir,
while subsequent runs will be incremental and take less time.
.Pp
.Dl $ smarc -m path/to/maildir -o path/to/outdir
.Pp
On multi-processor machines multiple jobs may be used to save some
time with the
.Xr smarc 1 Fl j No flag.
.Pp
The generated files may be compressed to save bandwidth:
.Pp
.Dl $ gzip -krq /var/www/smarc </dev/null 2>/dev/null
.Ss 5. Set up the database for searching
This is a suggested yet optional step.
.Pp
.Xr msearchd 8
offers full text search capabilities using a sqlite3 database that has to
be populated with
.Xr smingest 1 .
First, create a directory in the
.Pa /var/www
.Xr chroot 8
jail:
.Bd -literal -offset indent
# mkdir -p /var/www/msearchd
# chown smarc /var/www/msearchd
.Ed
.Pp
Then, populate the database with all emails in the maildir:
.Bd -literal -offset indent
$ sqlite3 /var/www/msearchd/mails.sqlite3 \e
	</usr/local/share/examples/smarc/schema.sql
$ mlist ~/Mail/smarc | smingest /var/www/msearchd/mails.sqlite3
.Ed
.Pp
At this point,
.Xr msearchd 8
can be started.
.Ss 6. Configure the web server
The web server needs to serve the contents of the outdir as-is and
handle the requests for
.Pa /search
via the
.Xr msearchd 8
FastCGI server.
A sample
.Xr httpd 8
configuration is provided here for reference:
.Bd -literal -offset indent
server "marc.example.com" {
	listen on * port 80
	root "/smarc"
	gzip-static

	# leave out when not using msearchd(8)
	location "/search" {
		fastcgi socket "/run/msearchd.sock"
	}
}
.Ed
.Sh HANDLING NEW MESSAGES
New messages should be fetched periodically using tools like
.Xr fdm 1
or
.Xr mbsync 1 ,
the database updated with
.Xr smingest 1
and the web archive refreshed using
.Xr smarc 1 .
.Pp
It is recommended to create a script like the following and schedule
its execution periodically with
.Xr cron 8 :
.Bd -literal -offset indent
#!/bin/sh

set -e
fdm -l fetch
minc ~/Mail/smarc | smingest /var/www/msearchd/mails.sqlite3
smarc
yes | gzip -krq /var/www/smarc/ 2>/dev/null || true
.Ed
.Pp
If
.Xr msearchd 8
is not used,
new messages still needs to be incorporated
.Po i.e.\& moved from
.Pa new/
to
.Pa cur/
.Pc
but no database has to be updated.
In that case simplify the
.Xr minc 1
invocation as:
.Pp
.Dl minc -q ~/Mail/smarc
.Pp
and don't call
.Xr smingest 1
at all.
.Sh HANDLING MULTIPLE MAILING LISTS
If the archive for multiple mailing lists needs to be served from the
same box, care must be taken to use different directories and database
files to avoid mixing messages.
.Pp
.Xr msearchd 8
handles only one database at a time, so multiple instances need to be
run, each pointing at the database for only one mailing list.
Different FastCGI socket path needs to be used per-instance.
.Pp
.Xr smarc 1
outdir, maildir and cachedir must be unique per-mailing list, i.e.\& the
.Fl c , Fl m No and Fl o
flag must always be provided.
.Pp
Very likely, each mailing list will needs its own set of templates, so
those needs to be prepared and both
.Xr smarc 1
and
.Xr msearchd 8
have to be pointed at the right template directory.
.Sh SEE ALSO
.Xr minc 1 ,
.Xr smarc 1 ,
.Xr smingest 1 ,
.Xr sqlite3 1 ,
.Xr httpd 8 ,
.Xr msearchd 8