treeinfo: add mirrorlist/metalink support

[jcpunk]

For setting up CentOS Variants we will need the ability to set mirrorlists for repository URLs. This PR explores what would be needed to get this first into .treeinfo support.

Remaining bits in this repo

• increment the .treeinfo version
• documentation
• unit tests

Didn't want to get too far before starting to chat on this....

[lubomir]

Can you explain more about how this is supposed to work? What and when is going to set these values? What is going to use them?

If you need it just for .treeinfo, why change composeinfo.json too?

[jcpunk]

My long term plan is to get variants for various deployment options (OpenStack, Ceph, oVirt, etc). Not all mirror sites are going to want to mirror all variants. I expect over time there will be a fair number. So, by having a mirror list or metalink we can give a consistent experience to users without requiring extra space from disinterested mirrors.

Eventually I'm going to build some tooling to set these values, but first I wanted to see about getting this idea through/evaluated/merged/etc.

I tried to keep .treeinfo and composeinfo.json in sync regarding fields they recognize.

[lubomir]

I'm still having trouble understanding how this will be used. Right now this code is used in Pungi when a compose is created. The files are written once. I don't see where mirrors come into play.

In Fedora use case, the finished compose is copied to mirrors, but there is nothing in the process that could update the files. Particularly .treeinfo may be baked into ISOs (for example there's a Fedora Server DVD that includes it). Who and when is going to fill in the data?

[jcpunk]

I'm coding up an anaconda extension that will permit switching between variants if multiple variants are present. The target media is really the netinstall.iso, but I've got plans to extend this further.[1]

For example:
You've got CentOS 8 netboot.iso, the .treeinfo on the selected mirror lists the following Variants:

• OpenStack
• oVirt
• CEPH

Anaconda will present an option in the UI to permit selecting one of these variants and populate the relevant bits.[2]

It makes the most sense to me from a programming perspective to get support into the metadata, extend anaconda with support, extend pungi.

Eventually I'd like to get this into Fedora so that you can flip the Workstation media into Server media (or Cloud or Atomic) with a simple click. But my real target is at getting CentOS SIGs easily exposed within the UI.

Right now I've got a tool to customize .treeinfo with the extra bits, but it is currently super ugly.

[1] Once I get the extension coded and whatnot, I plan to extend pungi to allow adding these extra bits to the media.
[2] I'm still coding this extension.

[lubomir]

How does netinst ISO plug into it? If it's reading the .treeinfo from the mirror, it has to already know where it is, so including this data changes nothing. Including .treeinfo on netinst ISO is not correct, since it doesn't actually contain a tree of packages.

In general I don't like this strong coupling between building and shipping. Given that the data will be mostly static (at least I hope so), it would seem preferable to me to put it into some RPM that will get to the netinst media. Anaconda could read it from there.

[jcpunk]

When anaconda fetches .treeinfo it is picking a specific variant (Workstation, Server, etc). In my planned usage, there will be some additional variants that may not be as widely mirrored (OpenStack, oVirt, etc). This would have the workflow of :

1. Connect to well known address
   Example: https://mirrors.centos.org/linux/7/$arch/os/
2. List variants advertised there
   OpenStack, oVirt
3. Permit user to select an advertised variant not mirrored at well known address
   OpenStack
4. Anaconda connects to new mirror location per selection
   Example: https://www.rdoproject.org/repos/7/os/$arch/

Various variants may be added over time and using the .treeinfo file as published with the release would permit adding these without reissuing media.

[jcpunk]

Any further thoughts on this? If approved I'd propose this for CentOS 8 as part of what I'm currently working on for Anaconda in CentOS 8.