RFC use JSON files from downloads.openwrt.org

[aparcar]

Hi, cool project!

After being tied to have ASU parse the download server, I started to add some hidden JSON files for initial testing. Maybe you could incorporate them in your tool to make things easier?

• Latest versions: https://downloads.openwrt.org/.versions.json
• Available targets: https://downloads.openwrt.org/releases/23.05.3/.targets.json
• Available packages: https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/.index.json

If these files happen to be of use, I may make them visible to everyone (i.e. removing the leading dot).

[dangowrt]

As the URL to asu should be configurable, the best would be if the URL-base for those JSON files would be part of overview.json on ASU.
In that way, forks would still work with owut as well and just supply their download server there.

[efahl]

Hi Paul, thanks for those links!

Latest version: Is this to be used by ASU to generate https://sysupgrade.openwrt.org/json/v1/overview.json ? That's the file I've found most useful for getting version info, as it's categorized nicely by branch. I seem to recall discussion about release-SNAPSHOT somewhere, did that get resolved as "don't advertise them?"

I'll dig through those others soon.

[efahl]

As the URL to asu should be configurable, the best would be if the URL-base for those JSON files would be part of overview.json on ASU.

How about add upstream_url or upstream_base or ??? to https://github.com/efahl/asu/blob/main/asu/common.py#L496 in the server section. I can submit a PR if that seems appropriate.

            "upstream_url": config["UPSTREAM_URL"],

Refs https://github.com/search?q=repo%3Aopenwrt%2Fasu%20UPSTREAM_URL&type=code

[dangowrt]

I strongly beliebe we should have exactly one URL configurable for the user. Everything else is error prone and confusing. @aparcar please make asu return the URL of the download server used in case it doesnt already do that.

[aparcar]

I can submit a PR if that seems appropriate.

Please do. I'm fine with the overview.json usage since I don't plan to drop that file, however I very much suggest to use upstream files wherever possible. I don't want to scrape all those package feeds all the time. Ideally the index.json files are you directly on the upstream server.

[efahl]

I'm picking through https://downloads.openwrt.org/.versions.json in earnest...

My current scheme is to scan ASU overview and build a little local data structure:

    "branches": {
        "21.02": {
            "latest": "21.02.7",
            "versions": [
                "21.02.6",
                "21.02.7",
                "21.02-SNAPSHOT"
            ]
        },
        "22.03": {
            "latest": "22.03.4",  <-- wrong, should be 22.03.5
            "versions": [
...

This is out-of-date with respect to latest on 22.03, and is missing several versions that are available from downloads.

So, I grab .versions.txt and augment the branches with any missing versions from it, updating latest. (I'm ignoring all the 19.07... and other versions that don't have a branch in ASU overview.)

    "branches": {
        "21.02": {
            "latest": "21.02.7",
            "versions": [
                "21.02.0",
... all of them
                "21.02.6",
                "21.02.7",
                "21.02-SNAPSHOT"
            ]
        },
        "22.03": {
            "latest": "22.03.5",  <-- fixed by getting newest non-snapshot from .versions
            "versions": [
...

I like it in that it reduces messing with ASU branches.yml to add new versions, as soon as they are built.

I have questions:

1. Will the -rc versions be included in .versions.txt? They are available on downloads assuming you know where to look, but not anywhere else (wait, the firmware selector harvests them in its collect.py). Seems like we definitely want those for beta testers...

2. Likewise for -SNAPSHOTs on release branches, which appear in the overview but not in .versions. These seem like they should be present for testers.

3. I see .versions.txt contains 23.05.1, the broken release that shouldn't appear in any upgrade list. Do we have an official mechanism for suppressing it?

4. Would there be any value in adding ancient branches (17.01-19.07), maybe with an owut --legacy option? I'm thinking "don't encourage them" applies here, so no.

[dangowrt]

however I very much suggest to use upstream files wherever possible.

In that case you should include a reference to the base URL (ie. https://downloads.openwrt.org/ somewhere in overview.json). Configuring more than one URL is a bad idea and will result in confusion and inconsistencies on the user end.

[efahl]

In that case you should include a reference to the base URL somewhere in overview.json

Yes, that's exactly what openwrt/asu#852 does.