Release and utilization planning

[BotBlake]

We are pleased to announce that the project is progressing well. The path to a first test release is getting shorter and shorter, and so detailed questions about the release process are slowly emerging.

---

Release Questions

Specifically, it still needs to be clarified HOW the project should be technically released. If you have any ideas here, please feel free to write them here.

CLI / Project Usage

A "usage concept" also still needs to be agreed upon.
Which CLI options should remain, how should the "developer mode" be called up, ...

Project Scope

Another slowly emerging topic is the project scope.
Which functions should be part of the project? Should the client's only goal be to perform the tests specified by the server?
The client could also be used as a standalone benchmark for individual files, or it could be integrated into other Python projects.
Please feel free to add your two cents

---

All these topics should be recorded here in the discussion if possible.
We thank you for your interest and collaboration.

[BotBlake]

It may be useful to talk about the desired user interface first. Here is a description of how the program currently runs/is operated.

CLI Arguments

--server {URL}:

• required
• web URL of the server
• In developer mode, a local .json file can also be used here as an alternative to the server. Since this (at least in its current form) is NOT suitable for the productive environment, it is hidden behind the developer mode

--gpu {idx}:

• optional
• Index of the GPU to use. The list is output when the script runs for the first time and recognizes multiple GPUs.
• If a 0 is passed here, GPU tests are skipped

--nocpu:

• optional
• If this option is set, all CPU tests are skipped.

--video {path}:

• optional
• the corresponding video files are searched for in this path, or downloaded into it if necessary. If it is not specified, the videos are stored in ./videos.

--ffmpeg {path}:

• optional
• The ffmpeg portable archive is installed/unpacked here. If nothing is specified, the path ./ffmpeg is used.

--output {path}:

• optional
• The output file is saved in ./output. If a Path is specified here, it will be saved in PATH/output.json instead

--debug:

• optional
• This disables the progress bar, behind which a lot of output is hidden. In addition, it activates the developer mode. By default, this does NOT change the output file, but additional options (such as the local .json file) are enabled.

Runtime Procedure

Initialization:

1. Query the server to determine which platforms are supported (platform token)
2. Hardware / operating system detection
   --> GPU Selector (if not bypassed by CLI option)
3. Load test data for corresponding platform from server

Downloading:

(the system always checks whether the files already exist and the checksums match)

4. Get ffmpeg archive OR unpack existing
5. Get test video files

Benchmark:

6. Query: Ready to start?
7. Start benchmark (no user input required)
8. Save output.json in corresponding output folder

[joshuaboniface]

re: the questions:

1. "HOW the project should be technically released" I think it should be packaged as a single standalone executable/script file that can be downloaded from the main repository server on demand, sort of like our Debuntu installer is.
2. "A "usage concept" also still needs to be agreed upon." I'd like to see release builds not feature a developer version, and run with as little human input as possible. So, in that sense, the fewer arguments and the more autodetection, the better. A built-time flag would be ideal for this: set it during production builds to disable stuff like setting the server URL (hardcode it to us) and handle download/upload (results) automatically, autodetect GPUs, etc.
3. " Which functions should be part of the project? Should the client's only goal be to perform the tests specified by the server?" Yes, i think the ultimate goal is to run standardized tests.

The ultimate goal is to replicate https://www.elpamsoft.com/?p=Plex-Hardware-Transcoding but for more than just nVidia GPU hardware, with the ultimate goal to provide raw data for someone looking to purchase hardware for Jellyfin transcoding. I'd envision us explicitly straying away from analysis of the data (e.g. "this is better than that") but simply provide the raw numbers and let users (or other writers, bloggers, etc.) perform the analysis. I don't believe expanding the scope to be a general "performance analysis" tool is idea, because that's going to just result in an explosion of features; it should be tailored specifically to our needs and nothing else.

[BotBlake]

Basically, the project seems to be going in exactly the right direction so far.
At the moment we have a “simple” python script, which can be installed via pip (wheel file) and is then directly available.

It is planned to integrate the URL of the server (hwa.jellyfin.org) as a default value in the --server option.
So it does not HAVE to be used, but CAN be used.

How the server evaluates the data is not part of the client. We only evaluate the data to the extent that it can be traced back to when and why a run failed and with how many workers it ran.
Does the current server-side visualization work reasonably well?

The runtime is currently (at least on my hardware) roughly within the time frame. Currently approx. 2 hours. There are currently “problems” with the NvEnc limit and the runtime (as the runs are scaled down again until the limit no longer occurs), but this can be solved.
We also have another (but complex) trick to reduce the runtime.

Otherwise, once again the question about the individual CLI options.
We have followed the “can, but don't have to” concept here.
So that a CLI option CAN be used, which then also removes all interactivity. If the CLI option is NOT set, the program interactively asks for

[mcarlton00]

Runtime is definitely one of the concerns i have. It takes ~40 minutes to run tests on my dev rig with a single gpu and the --nocpu flag. It may be worth looking at cutting the test files to be shorter if we can still get a valid test out of them being 10-15 seconds instead of 30.

[joshuaboniface]

I think we need 30s to be a valid test, as it can be a bit fast or slow at the start, and takes some time to really stabilize.

[BotBlake]

Venson had an awesome approach, where we start evaluating after 500 frames as usual, but instead of running until finished, we could simply limit it to lets say frame 600 and call it a day, when the data is "stable"

Also @mcarlton00 Do you have an NVIDIA GPU?
Runtime on non DriverPatches NVIDIA GPUs is horrible atm, because of the downscaling.

Other then that the runtime is somewhat on point for the GPU tests I think

[mcarlton00]

This test was on my AMD system. I do a nvidia gpu in my server that i haven't ran it against yet.

If the runtime is going to be measured in hours, that's gonna have to be communicated very clearly or else some users are probably gonna get a nasty surprise and we might not get as many results as we're hoping for when people don't let it finish.