Ubuntu package

[dstillman]

With the switch to a single standalone version, it's possible we should put out an official Ubuntu package. I have no idea what's involved with this, but https://forums.zotero.org/discussion/25317/install-zotero-standalone-from-ubuntu-linux-mint-ppa/p1 and https://github.com/smathot/zotero_installer are possibly relevant.

I'm not going to set this up (other than creating an official Zotero account somewhere if necessary), but if it can be made simple enough I'm happy to add it to the existing deploy scripts.

[dstillman]

Or, alternatively, package the Linux version using Flatpak or AppImage.

[nedrichards]

there's currently a flatpak package of this on flathub - although a native one from source code would be better. As per flathub policy, if you'd like to be responsible, as the developer you'd be more than welcome.

[Eothred]

I added a shell script that creates the appimage on a fork. It essentially depends on 'convert' (for the icon) and wget which are available on most systems I expect. It isn't very properly written (so I will not make a pull request of this in the current state). Someone could probably easily take those steps and write it in a way that it is more properly integrated into the zotero build system.

From the top of my head I know of canoe as another project using npm that builds appimages as releases. (Gruntfile? I know very little about this system in general, sorry) Maybe one can see how it is built there.

[howthebodyworks]

@retorquere maintains a repo of zotero debs.

[retorquere]

.deb packaging is easy if you have access to the files as laid out just before they'll be tarballed-up., and if you're OK with what is known as a "simple repo" my scripts are pretty stable now, but the way they package Zotero they're not likely to ever be mainlined into Debian (which is the usual path to get into derivative distros like Ubuntu) because I just put Zotero on disk as it is tarballed, and the zotero-standalone-build fetches a binary (firefox ESR) -- Debian wants everything to be either build from source, or built on an existing debian dependency.

PPA packaging is a mess and is hard to automate. The only benefit is that it's an easier route to get into Ubuntu, but they also want from-source builds, so see former point.

My scripts would be simple to adapt for packaging as part of the build process -- and they'd be stripped pretty far down as they'd not have to detect new versions being available, they wouldn't need to be for both Zotero and Juris-M, etc. I'd be happy to relinquish hosting the .debs.

The only real thing that would remain is where to host the binaries -- I host them in github releases, but hosting them on S3 would be simpler still. The most important driver for the decision on where to host them is whether you want download stats I think. GH stats are not super great, I don't know about S3. SourceForge is also super easy to set up, and has better download stats (and it's allowed to only use them as a bin hosting service), but SF isn't as popular with the OSS crowd as they once were.

Bintray seemed like the most logical choice but it was finicky to set up, and I hit upload limits during the first two weeks of testing, and I didn't want to be hobbled by the risk that I'd need to put out a release but couldn't.

[howthebodyworks]

@retorquere aside: Your various contributions to Zotero, especially BBT have saved days of my life, thank you.

Also, thanks for the high-speed explanation of debs, Emiliano! I wonder if we should be avoiding some of the friction that you identified by packaging it the way Ubuntu seems to want pre-built apps distributed these days, i.e. Snap packages. This process I think is not dissimilar to the flatpak builds, which one can also run on Ubuntu.

Snapcraft explicitly supports pre-built-binaries and seems to include hosting, so that sidesteps the hosting and build questions. It might have other frictions, however. I'm not sure if people are have strong commitments to having both formats/pros and cons?

[retorquere]

I have no strong opinion on snaps -- I tend to opt for debs when available but that's probably just because I'm used to them. To be clear, debs support pre-build binaries, it's just that PPAs/Debian build rules don't allow them (which is why e.g. Oracle Java is packaged as a shell script that downloads and installs Java during .deb installation).

Snap tools are likely to be more pleasant to work with, and I say this not because I know them but because I can't imagine them being equally frustrating as the .deb tools. It took me ages to find out what was in the end very simple; there's a thousand ways to package debs, and a lot of the docs and tutorials tend to

1. want to explain everything so even the most complex cases are covered ("If you wish to make an apple pie from scratch, you must first invent the universe"), and
2. be outdated

It is not a pleasant process, but once done, it's set and forget. Because I still want to verify I'm releasing the right thing I'm releasing "manually" but that's really just a commit of an auto-updated config file (and that would be skipped for a real Zotero build, because the check I do is just whether it indeed grabbed the correct version); Travis takes care of the actual build & repo update.

One upside of debs vs snaps is that Chromebook users can install those. Other than that, anything that reliably gets me the latest version of Zotero is fine by me. I have no preference one way or the other.

[retorquere]

I did have at one point a setup that used a fork of zotero-standalone-build to build a PPA-compatible package from scratch, but the PPA build infra blocks network access during build, so you can't fetch the firefox binary as part of the build.

Since you can't fetch binaries during build, and you're not allowed to have them as "source", PPAs/Debian mainline will just not work. The same goes for anything that relies on npm packages -- they must be existing Debian/Ubuntu packages. I don't see mainlining ever happening for this reason, because it will affect the electron build in the same way. (edit: although technically you could check in the node_modules directory to get around this, but that's yucky)

Self-hosted debs don't have this problem, and neither do snaps/flatpaks.

[retorquere]

I can convert (simplify, mainly) my existing script for https://github.com/retorquere/zotero-deb, but I'd need to know

• what is the build environment for releases? Packaging debs requires a debian-based system to create the packages. I currently build on Travis.
• what is the stage in the build where the client is laid out on disk as it is about to be tarballed and where is it laid out? Or should I just work from the tarball (not a problem, that's what I do now)
• are you OK with a "simple repo" built with apt-ftparchive?
• where do you intend to host the repo (S3, GH release, something else)?

(edit: downside of a simple repo is that there's no multiple versions for separate distro-versions, but Zotero doesn't have this in any case)

[dstillman]

what is the build environment for releases?

Not Debian-based, but we could use Docker for that part.

what is the stage in the build where the client is laid out on disk as it is about to be tarballed and where is it laid out?

Files are available in dist/ ($DEST_DIR) before the tarball step.

are you OK with a "simple repo" built with apt-ftparchive?

I'm not sure what that means.

where do you intend to host the repo (S3, GH release, something else)?

The repo in this case is…the thing that gets checked for updates? What form does that take? We tend to keep update manifests on actual webservers in case we need to do conditional updates for any reason (e.g., not serving an update to a particular set of clients). Downloadable files would go on S3 with the rest of the binaries. But I can handle adding the upload stuff as long as I know what files need to go where.

Thanks for working on this!

[retorquere]

Files are available in dist/ ($DEST_DIR) before the tarball step.

But they wouldn't be in the Docker env, right? The scripts inside docker could just fetch the tarball as I do now.

are you OK with a "simple repo" built with apt-ftparchive?

I'm not sure what that means.

It means that given a Zotero arch/version, there's only one .deb, not one for each Debian/Ubuntu/whatnot release. In a full repo, you can have one version for Ubuntu 18.10, one for 18.04, etc, but I tried setting that up and it was a major PITA. Given that Zotero only has a single build per arch/version, I don't see any benefit in trying.

where do you intend to host the repo (S3, GH release, something else)?

The repo in this case is…the thing that gets checked for updates?

Correct, and where the debs are downloaded from

What form does that take?

Any web server that supports https and that allows files to be downloaded directly (although 301/302 redirects are OK) with a GET;

https://hostname/whatever/path/here/can/be/long

will work as long as

https://hostname/whatever/path/here/can/be/long/InRelease
https://hostname/whatever/path/here/can/be/long/Packages.bz2

etc will work; eg, a simple website hosted in an S3 bucket will work

You can see a full list of assets at https://github.com/retorquere/zotero-deb/releases/tag/apt-get for a simple repo; obviously, a Zotero repo would not host Juris-M binaries. I'd still be maintaining the repo for JM binaries until Frank decides he wants to host them himself, but I'd strip Zotero from the repo.

We tend to keep update manifests on actual webservers in case we need to do conditional updates for any reason (e.g., not serving an update to a particular set of clients).

A particular set of clients as in OS-dependent? Or another discriminator?

Downloadable files would go on S3 with the rest of the binaries.

A redirect to S3 would be OK (technically the assets on GH releases are hosted from S3 buckets, and you get sent there by a redirect), but simple repos (and as far as I can tell, full repos) require all URLs to sit on the same base; you can't have just https://hostname1/Packages and https://hostname2/zotero_5.0.73-1_amd64.deb

(the -1 at the end is a version bump so I can fix things in the packaging while keeping the binaries at the same version).

[dstillman]

But they wouldn't be in the Docker env, right?

No, but we could easily specify the dist directory as a mounted volume. While we could run this as a separate step after the normal tarball upload, there's no real reason to redownload and extract the same files.

Given that Zotero only has a single build per arch/version, I don't see any benefit in trying.

Right.

A particular set of clients as in OS-dependent? Or another discriminator?

OS version or current Zotero version, mainly. Though I doubt we'd care about distro/version here, and presumably we wouldn't get the Zotero version anyway. So this probably doesn't really matter.

(Basically, this has been useful in the past for things like not serving Zotero above a certain version to users on old versions of macOS, or serving a special manual-upgrade message to clients where the updater was broken.)

[retorquere]

Does the build process build both 32 and 64 bit? To finalize the repo you need all packages in place.

[retorquere]

It's possible to run them one by one no issue, but

1. You'd need to have a staging area where they come together, and
2. You don't want the repo-refresh after the package build to be done in parallel.

[retorquere]

Can I detect in the dist directory what architecture (32/64 bit) was being built?

[retorquere]

(other than by inspecting the ELF header of zotero-bin)

[dstillman]

After the done here, there'll be Zotero_linux-i686 and Zotero_linux-x86_64 directories.

[retorquere]

OS version or current Zotero version, mainly. Though I doubt we'd care about distro/version here, and presumably we wouldn't get the Zotero version anyway. So this probably doesn't really matter.

(Basically, this has been useful in the past for things like not serving Zotero above a certain version to users on old versions of macOS

I hate to say this, but this is possible using a full repo. You can have varying versions per target distro. It was a major PITA to get even halfway though, I didn't get it to work last time, and the docs on the process are not set up for easy digestion. Tutorials abound, many of them outdated and conflicting.

or serving a special manual-upgrade message to clients where the updater was broken.)

Broken updaters are out of scope for debs -- Zotero-updating is disabled in the debs because they're installed as root, updates would come from the dep repo.

After the done here, there'll be Zotero_linux-i686 and Zotero_linux-x86_64 directories.

Ah but that script has $arch so it could just pass that in.

[dstillman]

I hate to say this, but this is possible using a full repo.

That's OK. There's essentially no chance we'd worry about old distro compatibility the way we've had to worry about macOS and Windows compatibility.

Broken updaters are out of scope for debs

Right.

If we do decide to split things up and use redirects to S3 for the debs, I'll deal with it. For now we can assume they'll just go to the same place.

[retorquere]

I hate to say this, but this is possible using a full repo.

That's OK. There's essentially no chance we'd worry about old distro compatibility the way we've had to worry about macOS and Windows compatibility.

You have no idea what a relief this is 😆

If we do decide to split things up and use redirects to S3 for the debs, I'll deal with it. For now we can assume they'll just go to the same place.

That's cool. Then it's mostly done -- at the done point there's Zotero_linux-i686 and Zotero_linux-x86_64? Not "or"?

edit: I see now, it is indeed "and"

[retorquere]

Alright, the current script can be started after done; when done, it will have apt/repo ready to upload. a GPG key needs to be present, I've generated mine using this.

[retorquere]

"the current script" being #73

[dstillman]

Resurrecting this in light of https://forums.zotero.org/discussion/comment/394321/#Comment_394321

What's the current feasibility of just integrating the .deb build step into our Linux build script?

We run the build script for Linux releases manually in a Linux (but non-Debian-based) environment.

[retorquere]

As long as either the tarball or the layout pre-tar is available, it can create the package. The main question is whether it is desired to retain older debs. When a new deb is added, the old debs must be present to rebuild the index. I do that now by downloading the release when a new build is needed.

Another question is version management. De debs follow the zotero release of course, but if a problem is found in the packaging itself, I add an extra sub-minor component to keep the zotero release number while still promoting apt to upgrade. These sub-minors are semiautomatic. I can explain this when we have the infra in place.

I'm in the last stages of a major simplification of the build script, which should be done in the weekend.

[retorquere]

A debian environment is required but that is easily done in a github action (which is where it runs now).

[dstillman]

The main question is whether it is desired to retain older debs. When a new deb is added, the old debs must be present to rebuild the index. I do that now by downloading the release when a new build is needed.

Yes, our packaging scripts do something similar to generate binary diffs. There's a cache folder of recent downloads, and if they're missing it will download them based on an index of recent release versions and a specified number of versions to support. But what's the practical upshot in this case? These are the older versions that people can install manually by passing a different version number to dpkg/apt? And if they're not present those versions become no longer available for manual installs?

A debian environment is required but that is easily done in a github action (which is where it runs now).

We'd have to do it in Docker. We do tests via CI but not builds.

[retorquere]

Yes, our packaging scripts do something similar to generate binary diffs. There's a cache folder of recent downloads, and if they're missing it will download them based on an index of recent release versions and a specified number of versions to support. But what's the practical upshot in this case? These are the older versions that people can install manually by passing a different version number to dpkg/apt? And if they're not present those versions become no longer available for manual installs?

Correct. My assumption is that people might sometimes want to downgrade for diagnosis / emergencies. But if not present during the repo build, they'd be unavailable to apt; if the debs are available elsewhere they could be installed using dpkg.

We'd have to do it in Docker. We do tests via CI but not builds.

Docker would work.

[dstillman]

OK, happy to look at whatever you have after you're done working on it. Would be nice to make these part of the official release process.

[retorquere]

Sorry we haven't gotten to this yet. Re: your question in the forums, if it's just a question of cost, we can certainly create an S3 bucket (and CloudFront distribution) for you, but if we're planning to take these over eventually, is that just an extra manual change that people will need to make? Or is there a smooth migration process, assuming the bucket stays the same?

That depends under which URL the files are available. If eg a cloudfront redirect/proxy were put in place, it would just mean the cloudfront redirect would be changed, not the URL, and that would be seamless for the user. If the URL changes, action is needed from the user to change along.

(Or maybe this is ready-to-go enough that it's something I can just set up in an hour?)

that depends on the environment in which the script is ran. I have my current repo refresh broken up in two parts:

1. One part that prepares the environment; arranges that the existing repo content is present, and that any new Zotero versions that must be added are laid out on disk in the same structure that is in the tarball.
2. One part that, given the environment on disk from 1. rebuilds the laid-out client and refreshes the repo

at which point part 1 detects that changes have been made and updates the hosting.

If this is to be ran from the Zotero build, you'd only need part 2, and that is ready-to-run with some configuration only. If the Zotero build environment takes care of part 1, that should be all.

---

Another way to set this up would be to use the gh cli to just do gh release download to the Zotero hosting from the github repo, where the Zotero-hosted apt repo would just be a mirror (but the primary source for users) of the gh release. I could transfer my current repo to Zotero ownership, keep doing the upkeep, and the two infrastructures would be safely separated. I would bring down the sourceforge mirror and rename the GH release, which would alert existing users the hosting needs changes, and after they make the change, it would be Zotero-hosted.

[retorquere]

It did turn out that I had just misconfigured b2 + cloudfront, so it should be possible for me to set up a reliable repo that way with minimal cost through the bandwidth alliance.

[dstillman]

I'm seeing if I can get the build script to run via Docker.

[retorquere]

You will need to have a gpg key in the docker environment to sign the packages (and have it listed in config.ini)

[dstillman]

Does anything actually check the signature?

Also, what's the deal with this in config.ini?

5.0.96.2 = 3
5.0.96.3 = 3
5.0.97.57+07df7d0de = 3

[dstillman]

Looks like Release.gpg is maybe checked by apt? I assume that's created here using the same configured key?

Are the .deb signatures actually checked?

[retorquere]

Does anything actually check the signature?

the apt toolchain does, yes. Nothing during build.

Also, what's the deal with this in config.ini?

5.0.96.2 = 3
5.0.96.3 = 3
5.0.97.57+07df7d0de = 3

In the case a problem is found in the packaging process itself after the debs have already been made available, I need to be able to generate a new package-version from the same zotero-version. In the case of 5.0.96.1, the package version is also 5.0.96.1, but for 5.0.96.2 there were 5.0.96.2, 5.0.96.2-1, 5.0.96.2-2 and finally 5.0.96.2-3, and apt will have seen these different versions and have upgraded to them. These entries are only present as a sample, I didn't really find 6 issues, but I needed to test the functionality.

Looks like Release.gpg is maybe checked by apt? I assume that's created here using the same configured key?

correct.

Are the .deb signatures actually checked?

Yep, by apt.

[retorquere]

I think I have the B2 hosting through the bandwith alliance set up correctly now, so I could in principle switch the hosting to that, and it would be at no cost.

I'm keeping the B2 repo fresh right now, I am not yet announcing it because I don't want to have the users switch too often, but the sourceforge mirror system isn't great -- every few days the automated tests fail because of it without any changes on my end, and that means users are hitting this too. GH releases are stable but require a patched client that certainly ESR users won't have. B2/S3 solves these issues.

BTW, I don't know how much hosting runs for Zotero, but if it's substantial, the bandwidth alliance could be worth looking into.

[dstillman]

Meaning you can use CloudFlare's CDN on their free plan and not pay for egress from B2? I couldn't find anything on the CloudFlare pricing page that actually discussed data transfer limits…

I still need to set up GPG but we can probably take this over.

Before we commit to this, how unusual is what we're doing here? How likely is it that something's going to change in Ubuntu that breaks upgrading from existing installs? This issue aside, that's happened at least once before, right? I pretty much only ever use yum-based distros, and custom yum repos are fairly common, but I'm not clear on whether this is as straightforward as those.

[retorquere]

Meaning you can use CloudFlare's CDN on their free plan and not pay for egress from B2?

That's my understanding, yes, and since on B2 you don't pay for ingress, the only charged actions are authentication and listing the contents of the bucket, but for me that falls withing the free daily allotment.

I couldn't find anything on the CloudFlare pricing page that actually discussed data transfer limits…

I've been looking for a while but haven't found any small print. The reason I was getting charged earlier was that for each rebuild, I was downloading the repo outside the bandwith alliance path, and while I was developing this I did that frequently.

I still need to set up GPG but we can probably take this over.

Cool.

Before we commit to this, how unusual is what we're doing here? How likely is it that something's going to change in Ubuntu that breaks upgrading from existing installs?

This repo setup has been in Debian since the beginning as far as I can tell, but outside the Ubuntu's PPA system there hasn't really been a strong culture of outside repo's in Debian, so documentation on it was pretty blurry. What has changed is where it is recommended the public key is stored on the users system. I don't see that changing anymore since each repo now uses its own keychain, the use of the single keychain was the (potential) problem. The current install.sh simply overwrites the keychain, so after they've switched to the zotero-hosted repo, my public key will be gone from their system.

This issue aside, that's happened at least once before, right? I pretty much only ever use yum-based distros, and custom yum repos are fairly common, but I'm not clear on whether this is as straightforward as those.

I haven't looked into yum repos but was actually thinking about creating RPM packages. But apt repos aren't all that hard in the end. The storage of the key for all (including the official) repos was an issue, but that's no longer in the install.sh

[retorquere]

Any news?

[retorquere]

Any news on this?

[retorquere]

If it's unlikely that this is happening soon(ish), I need to finish setting up the b2-based repo and move everything to that. Sourceforge is frustratingly spotty, and github runs into a debian problem that's not going to be solved quickly; nothing in debian happens quickly. I'd rather not have the users have to move another time when the Zotero repo is live, but sourceforge is unworkable.

[retorquere]

I'm going to prep migration to b2. I really can't wait any longer. It's disruptive to these users, and it weighs on me I don't have a solution for them.

[retorquere]

What environment does the build run in? Not debian-based, but something else then? Maybe I can get the required tools to work in the actual build environment. I may not need to get familiar with docker.

[dstillman]

OK, took some work, but I've got what appears to be a successfully signed repo via Docker.

Need to do some more refinement and testing, but I think this will work. More to come.

[dstillman]

Can you explain rebuild.py?

Do I need that, or can I just hard-code url and codename in install.sh and put it on a server? And then run build.py and an upload script as part of the build process?

[dstillman]

Other questions:

1. There's no point to wwwroot and repo being separate anymore, right? It's just putting files in repo?

2. zotero.gpg vs. zotero-archive-keyring.gpg — any reason to do the latter rather than the former? I see that Ubuntu has an ubuntu-archive-keyring.gpg.

[dstillman]

3. Is it expected that I had to run apt install ca-certificates to get it to accept the repo? Without that it was saying the certificate for www.zotero.org is expired, which it obviously isn't.

[dstillman]

4. I think it makes more sense to use separate repos for release and beta — if you do apt search zotero you really shouldn't see zotero-beta. Can I just make repo the channel?

[dstillman]

5. What version of Debian/Ubuntu have you been building on?

I used FROM ubuntu:22.04 for the Dockerfile, and then when trying to install the package on Ubuntu 20.04.2 LTS I got this:

[ubuntu:~] dan$  sudo apt install zotero-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 zotero-dev : Depends: libgdk-pixbuf-2.0-0 but it is not installable
E: Unable to correct problems, you have held broken packages.

This is the output of apt-cache depends firefox-esr in the Docker container:

firefox-esr
  Depends: lsb-release
  Depends: libatk1.0-0
  Depends: libc6
  Depends: libcairo-gobject2
  Depends: libcairo2
  Depends: libdbus-1-3
  Depends: libdbus-glib-1-2
  Depends: libfontconfig1
  Depends: libfreetype6
  Depends: libgcc-s1
  Depends: libgdk-pixbuf-2.0-0
  Depends: libglib2.0-0
  Depends: libgtk-3-0
  Depends: libharfbuzz0b
  Depends: libpango-1.0-0
  Depends: libpangocairo-1.0-0
  Depends: libpangoft2-1.0-0
  Depends: libstdc++6
  Depends: libx11-6
  Depends: libx11-xcb1
  Depends: libxcb-shm0
  Depends: libxcb1
  Depends: libxcomposite1
  Depends: libxcursor1
  Depends: libxdamage1
  Depends: libxext6
  Depends: libxfixes3
  Depends: libxi6
  Depends: libxrender1
  Depends: libxt6
  Recommends: xul-ext-ubufox
  Recommends: libcanberra0
  Recommends: libdbusmenu-glib4
  Recommends: libdbusmenu-gtk3-4
  Suggests: fonts-lyx

I'm trying again with FROM ubuntu:20.04, but isn't this sort of a problem in general? The current tarball is based on Firefox 60 ESR, but this is using the dependencies of firefox-esr 91.6.0esr+build1-0ubuntu0.22.04.1. How would that ever work?

Seems like the best we can do is hard-code a few things that will hopefully be available on most systems, no? (And then work on getting on a newer Firefox as soon as Z6 is out…)

[dstillman]

Oh, but after building on 20.04, it works. And the package name for libgdk-pixbuf is slightly different:

20.04:

Depends: gnupg, libatk1.0-0, libc6, libcairo-gobject2, libcairo2, libdbus-1-3, libdbus-glib-1-2, libfontconfig1, libfreetype6, libgdk-pixbuf2.0-0, libglib2.0-0, libgtk-3-0, libharfbuzz0b, libpango-1.0-0, libpangocairo-1.0-0, libpangoft2-1.0-0, libstdc++6, libx11-6, libx11-xcb1, libxcb-shm0, libxcb1, libxcomposite1, libxcursor1, libxdamage1, libxext6, libxfixes3, libxi6, libxrender1, libxt6

22.04:

Depends: gnupg, libatk1.0-0, libc6, libcairo-gobject2, libcairo2, libdbus-1-3, libdbus-glib-1-2, libfontconfig1, libfreetype6, libgdk-pixbuf-2.0-0, libglib2.0-0, libgtk-3-0, libharfbuzz0b, libpango-1.0-0, libpangocairo-1.0-0, libpangoft2-1.0-0, libstdc++6, libx11-6, libx11-xcb1, libxcb-shm0, libxcb1, libxcomposite1, libxcursor1, libxdamage1, libxext6, libxfixes3, libxi6, libxrender1, libxt6

Seems like they fixed the package name? Hopefully in a way where depending on libgdk-pixbuf2.0-0 still works in 22.04?

[retorquere]

Can you explain rebuild.py?

Do I need that, or can I just hard-code url and codename in install.sh and put it on a server? And then run build.py and an upload script as part of the build process?

Yes. rebuild.py is irrelevant to zotero hosting, it's just something I used to prep the environment for build.py in the github action. The contents used to sit in one script, I split it into two specifically to lift out the zotero-hosting-relevant bits.

1. There's no point to `wwwroot` and `repo` being separate anymore, right? It's just putting files in `repo`?

That depends on how the build environment is laid out, and what you want to have show up as the channel. repo describes the build environment, wwwroot describes the hosting environment, the difference between those two shows up (a.o.) in the Packages file and as the channel in the zotero.list. If the two are the same you don't need them separately, but if they're the same, you'll get the ./ in the zotero.list, and you indicated you don't want that (for good reason I think). If you don't want ./ in the zotero.list, you need these two paths (or something similar).

2. `zotero.gpg` vs. `zotero-archive-keyring.gpg` — any reason to do the latter rather than the former? I see that Ubuntu has an `ubuntu-archive-keyring.gpg`.

I saw a recommendation for the latter option somewhere in de debian discussion list. I don't recall where, sorry. There is no material difference between these two.

3. Is it expected that I had to run apt install ca-certificates to get it to accept the repo? Without that it was saying the certificate for www.zotero.org is expired, which it obviously isn't.

It is unexpected to me, and since I run this script every two hours in a github action, it doesn't seem like it's something needed structurally. I'd venture to guess that this has to do with the distro picked for the docker image.

4. I think it makes more sense to use separate repos for release and beta — if you do apt search zotero you really shouldn't see zotero-beta. Can I just make repo the channel?

I'm not sure why you shouldn't see that? If I issue apt search zo I get a heap of hits across repos. But yes, if that is your preference, you can make repo the channel by a combination of wwwroot, repo, and which directory you pass to build.py. I would prefer to do some testing on my end to make sure it works (and give directions on how).

5. What version of Debian/Ubuntu have you been building on?

Ubuntu 20.04.3 LTS (github actions) and Ubuntu 21.10 (my own system)

I used FROM ubuntu:22.04 for the Dockerfile, and then when trying to install the package on Ubuntu 20.04.2 LTS I got

22.04 is not out yet though

[ubuntu:~] dan$ sudo apt install zotero-dev

this is the beta?

The following packages have unmet dependencies:
zotero-dev : Depends: libgdk-pixbuf-2.0-0 but it is not installable
E: Unable to correct problems, you have held broken packages.

This is the output of `apt-cache depends firefox-esr` in the Docker container:

This is indeed picked up in build.py , but it's strange that firefox-esr lists it as a dependency when it is not available in that environment. and would indeed show up as an error on earlier distros.

I'm trying again with FROM ubuntu:20.04, but isn't this sort of a problem in general? The current tarball is based on Firefox 60 ESR, but this is using the dependencies of firefox-esr 91.6.0esr+build1-0ubuntu0.22.04.1. How would that ever work?

It will work as long as the build environment has the same ESR as zotero uses. But I can disable this automatic pickup and add the required packages to the (existing) list of explicit packages.

Seems like the best we can do is hard-code a few things that will hopefully be available on most systems, no? (And then work on getting on a newer Firefox as soon as Z6 is out…)

That will work.

Seems like they fixed the package name? Hopefully in a way where depending on libgdk-pixbuf2.0-0 still works in 22.04?

They sometimes include compatibility packages, but I can't say whether that will happen here. If there is no compat package, I can deal with it in a postinst, which is sort of unfortunate, but it will work. Is firefox-esr installable in 22.04?

[retorquere]

How did you run this BTW? I have Docker installed but how to use it is still pretty fuzzy to me. If I can replicate the run steps it'd be easier for me to change build.py to accomodate what Zotero needs; it will also make it possible to test for the differences in 22.04.

[retorquere]

WRT to the differences between 20.10/20.04 and 22.04, another option than using a postinst is to just assume that the required libraries are already installed on the system, or just to assume that libgdk-pixbuf already is installed . This is implicitly used for the official tarball. I just preferred the dependencies to be explicit.

[dstillman]

Thanks for the responses. I'll let you know if I have other questions, but you don't need to do anything else here — I've already made lots of changes for our purposes, and just need to do some more testing.

[retorquere]

In light of this, a postinst could detect that the package is being installed on ChromeOS and could perhaps set connector.url to http://127.0.0.1:8080/.

[dstillman]

(Haven't forgotten about this. Couldn't get to it for Zotero 6, but should be soon.)

[retorquere]

I have in the meantime gotten the backblaze hosting up and running; I'm deprecating the GH-release and the SF mirror (although I will keep them up), but for the moment, I'm good.

Is there interest in the postinst detecting/adjusting for ChromeOS? Or is that best left to the user?

[eebrown]

Sorry if this is only tangentially related now, but I just ran deb2npm to see which node dependencies are already packaged and which are missing from Debian, which is relevant for any efforts/interest in packaging Zotero within Debian. Results below:

Dependencies:
NPM                                               Debian
zotero (5.0.0)                                    None
├─ @citeproc-rs/wasm (^0.2.0)                     None
├─ ace-builds (^1.4.12)                           None
├─ bluebird (^3.5.1)                              node-bluebird (3.7.2+dfsg1+~3.5.36-1)
├─ classnames (^2.2.6)                            node-classnames (2.3.1-3)
├─ monaco-editor (^0.15.6)                        None
├─ prop-types (^15.8.0)                           node-prop-types (15.7.2+~15.7.4-2)
├─ react (^17.0.2)                                node-react (17.0.2+dfsg+~cs106.66.62-1)
├─ react-autosuggest (^10.1.0)                    None
│  ├─ es6-promise (^4.2.8)                        node-es6-promise (4.2.8-10)
│  ├─ prop-types (^15.7.2)                        node-prop-types (15.7.2+~15.7.4-2)
│  ├─ react-themeable (^1.1.0)                    None
│  │  └─ object-assign (^3.0.0)                   node-object-assign (4.1.1-6)
│  ├─ section-iterator (^2.0.0)                   None
│  └─ shallow-equal (^1.2.1)                      None
├─ react-dom (^17.0.2)                            None
│  ├─ loose-envify (^1.1.0)                       node-loose-envify (1.4.0+dfsg1-2)
│  ├─ object-assign (^4.1.1)                      node-object-assign (4.1.1-6)
│  └─ scheduler (^0.20.2)                         None
│     ├─ loose-envify (^1.1.0)                    node-loose-envify (1.4.0+dfsg1-2)
│     └─ object-assign (^4.1.1)                   node-object-assign (4.1.1-6)
├─ react-intl (^3.4.0)                            None
│  ├─ @formatjs/ecma402-abstract (1.11.3)         None
│  │  ├─ @formatjs/intl-localematcher (0.2.24)    None
│  │  │  └─ tslib (^2.1.0)                        node-tslib (2.3.1-2)
│  │  └─ tslib (^2.1.0)                           node-tslib (2.3.1-2)
│  ├─ @formatjs/icu-messageformat-parser (2.0.18) None
│  │  ├─ @formatjs/ecma402-abstract (1.11.3)      None
│  │  ├─ @formatjs/icu-skeleton-parser (1.3.5)    None
│  │  │  ├─ @formatjs/ecma402-abstract (1.11.3)   None
│  │  │  └─ tslib (^2.1.0)                        node-tslib (2.3.1-2)
│  │  └─ tslib (^2.1.0)                           node-tslib (2.3.1-2)
│  ├─ @formatjs/intl (2.1.0)                      None
│  │  ├─ @formatjs/ecma402-abstract (1.11.3)      None
│  │  ├─ @formatjs/fast-memoize (1.2.1)           None
│  │  │  └─ tslib (^2.1.0)                        node-tslib (2.3.1-2)
│  │  ├─ @formatjs/icu-messageformat-parser (2.0.18)None
│  │  ├─ @formatjs/intl-displaynames (5.4.2)      None
│  │  │  ├─ @formatjs/ecma402-abstract (1.11.3)   None
│  │  │  ├─ @formatjs/intl-localematcher (0.2.24) None
│  │  │  └─ tslib (^2.1.0)                        node-tslib (2.3.1-2)
│  │  ├─ @formatjs/intl-listformat (6.5.2)        None
│  │  │  ├─ @formatjs/ecma402-abstract (1.11.3)   None
│  │  │  ├─ @formatjs/intl-localematcher (0.2.24) None
│  │  │  └─ tslib (^2.1.0)                        node-tslib (2.3.1-2)
│  │  ├─ intl-messageformat (9.11.4)              None
│  │  │  ├─ @formatjs/ecma402-abstract (1.11.3)   None
│  │  │  ├─ @formatjs/fast-memoize (1.2.1)        None
│  │  │  ├─ @formatjs/icu-messageformat-parser (2.0.18)None
│  │  │  └─ tslib (^2.1.0)                        node-tslib (2.3.1-2)
│  │  └─ tslib (^2.1.0)                           node-tslib (2.3.1-2)
│  ├─ @formatjs/intl-displaynames (5.4.2)         None
│  ├─ @formatjs/intl-listformat (6.5.2)           None
│  ├─ @types/hoist-non-react-statics (^3.3.1)     None
│  │  ├─ @types/react (*)                         None
│  │  │  ├─ @types/prop-types (*)                 None
│  │  │  ├─ @types/scheduler (*)                  None
│  │  │  └─ csstype (^3.0.2)                      node-csstype (3.0.11-1)
│  │  └─ hoist-non-react-statics (^3.3.0)         None
│  │     └─ react-is (^16.7.0)                    None
│  ├─ @types/react (16 || 17)                     None
│  ├─ hoist-non-react-statics (^3.3.2)            None
│  ├─ intl-messageformat (9.11.4)                 None
│  └─ tslib (^2.1.0)                              node-tslib (2.3.1-2)
├─ react-select (^3.0.8)                          None
│  ├─ @babel/runtime (^7.12.0)                    node-babel (6.26.0+repack-3~bpo10+1)
│  ├─ @emotion/cache (^11.4.0)                    None
│  │  ├─ @emotion/memoize (^0.7.4)                None
│  │  ├─ @emotion/sheet (^1.1.0)                  None
│  │  ├─ @emotion/utils (^1.0.0)                  None
│  │  ├─ @emotion/weak-memoize (^0.2.5)           None
│  │  └─ stylis (4.0.13)                          None
│  ├─ @emotion/react (^11.1.1)                    None
│  │  ├─ @babel/runtime (^7.13.10)                node-babel (6.26.0+repack-3~bpo10+1)
│  │  ├─ @emotion/babel-plugin (^11.7.1)          None
│  │  │  ├─ @babel/helper-module-imports (^7.12.13)None
│  │  │  │  └─ @babel/types (^7.16.7)             node-babel (6.26.0+repack-3~bpo10+1)
│  │  │  ├─ @babel/plugin-syntax-jsx (^7.12.13)   node-babel (6.26.0+repack-3~bpo10+1)
│  │  │  ├─ @babel/runtime (^7.13.10)             node-babel (6.26.0+repack-3~bpo10+1)
│  │  │  ├─ @emotion/hash (^0.8.0)                None
│  │  │  ├─ @emotion/memoize (^0.7.5)             None
│  │  │  ├─ @emotion/serialize (^1.0.2)           None
│  │  │  │  ├─ @emotion/hash (^0.8.0)             None
│  │  │  │  ├─ @emotion/memoize (^0.7.4)          None
│  │  │  │  ├─ @emotion/unitless (^0.7.5)         None
│  │  │  │  ├─ @emotion/utils (^1.0.0)            None
│  │  │  │  └─ csstype (^3.0.2)                   node-csstype (3.0.11-1)
│  │  │  ├─ babel-plugin-macros (^2.6.1)          None
│  │  │  │  ├─ @babel/runtime (^7.12.5)           node-babel (6.26.0+repack-3~bpo10+1)
│  │  │  │  ├─ cosmiconfig (^7.0.0)               node-cosmiconfig (7.0.1+ds1-1)
│  │  │  │  └─ resolve (^1.19.0)                  node-resolve (1.20.0+~cs5.27.9-1)
│  │  │  ├─ convert-source-map (^1.5.0)           node-convert-source-map (1.8.0+~1.5.2-1)
│  │  │  ├─ escape-string-regexp (^4.0.0)         node-escape-string-regexp (4.0.0-2)
│  │  │  ├─ find-root (^1.1.0)                    None
│  │  │  ├─ source-map (^0.5.7)                   node-source-map (0.7.0++dfsg2+really.0.6.1-9)
│  │  │  └─ stylis (4.0.13)                       None
│  │  ├─ @emotion/cache (^11.7.1)                 None
│  │  ├─ @emotion/serialize (^1.0.2)              None
│  │  ├─ @emotion/utils (^1.1.0)                  None
│  │  ├─ @emotion/weak-memoize (^0.2.5)           None
│  │  └─ hoist-non-react-statics (^3.3.1)         None
│  ├─ @types/react-transition-group (^4.4.0)      None
│  │  └─ @types/react (*)                         None
│  ├─ memoize-one (^5.0.0)                        None
│  ├─ prop-types (^15.6.0)                        node-prop-types (15.7.2+~15.7.4-2)
│  └─ react-transition-group (^4.3.0)             node-react-transition-group (4.4.2+~cs4.4.4-1)
└─ url (^0.11.0)                                  node-url (0.11.0-5)

Build dependencies:
NPM                                               Debian
@babel/cli (^7.16.0)                              node-babel (6.26.0+repack-3~bpo10+1)
@babel/core (^7.16.5)                             node-babel (6.26.0+repack-3~bpo10+1)
@babel/eslint-parser (^7.16.5)                    None
@babel/eslint-plugin (^7.16.5)                    None
@babel/plugin-proposal-class-properties (^7.16.5) None
@babel/plugin-proposal-object-rest-spread (^7.16.5)None
@babel/plugin-proposal-optional-chaining (^7.16.5)None
@babel/plugin-proposal-unicode-property-regex (^7.16.5)None
@babel/preset-react (^7.16.5)                     node-babel (6.26.0+repack-3~bpo10+1)
@zotero/eslint-config (^1.0.6)                    None
babel-plugin-transform-es2015-modules-commonjs (^6.26.2)None
browserify (^14.5.0)                              node-browserify (17.0.0+ds-3)
chai (^4.1.2)                                     node-chai (4.3.6~ds1+~cs9.7.12-3)
chai-as-promised (^7.1.1)                         node-chai-as-promised (7.1.1+~cs7.1.4-2)
chokidar (^2.1.2)                                 node-chokidar (3.5.3-1)
co-mocha (^1.2.1)                                 None
colors (^1.4.0)                                   colors.js (1.4.0-3)
eslint (^8.5.0)                                   None
eslint-plugin-react (^7.28.0)                     None
fs-extra (^3.0.1)                                 node-fs-extra (10.0.1+~9.0.13-2)
globby (^6.1.0)                                   node-globby (12.0.2+~cs16.24.35-3)
jspath (^0.4.0)                                   None
mocha (^3.5.3)                                    node-mocha (9.2.2+ds1+~cs28.3.8-1)
multimatch (^2.1.0)                               node-multimatch (5.0.0-2)
react-virtualized (^9.22.3)                       None
sass (^1.28.0)                                    None
sinon (^7.3.2)                                    node-sinon (13.0.1+ds+~cs71.22.21-2)
universalify (^0.1.1)                             node-universalify (2.0.0-3)