Gradio UI

[ljleb]

Add a gradio ui to start the script from within the webui. I'll add a preview of what it looks like when done with the location of the settings in the page.

As of right now, it does not improve the experience by much for new users, but at least you can click on buttons now instead of having to use the command line.

Remains to be done:

• rearrange the components into a compact cols/rows structure
• display the nice unet weights chart at the end of the optimiser process
• manual test to make sure I didn't miss something (especially when calling the script)

We should consider adding a (sub-)tab to display arbitrary unet merge results without having to merge. I think it is a bit out of scope of this PR, let me know if you want me to add that.

Feel free to request changes or edit this PR.

Closes #9

[s1dlx]

awesome, thanks!

some thoughts:

• will this be optional or completely replace cli? I'd like to keep the core logic out of the gui code as much as possible
• if cli is not used anymore, can we make the gui read params from config files?
• I like the idea of showing the UNET in the GUI, perhaps the viz could be updated while the optimisation runs too

looking forward to see how it looks!

[ljleb]

will this be optional or completely replace cli? I'd like to keep the core logic out of the gui code as much as possible

My intention is to provide an additional way of using the optimisers. We can remove the cli if you want, but this PR does not change this part of the code in any way so far, except for sharing defaults and limited choices. (i.e. safetensors vs ckpt for --best_format)

At the moment the ui simply forwards the specified values to the cli script, it does not remove it or change its internal behaviour. It is called here:

sd-webui-bayesian-merger/scripts/optimiser_gui.py

Line 208 in 11101f1

process = subprocess.Popen(cli_args, cwd=script_root, stdout=sys.stdout, stderr=sys.stderr)

Maybe this is not necessary, I could make a direct call to bayesian_merger.main as an alternative.

I tried to keep the ui code as tidy as I could. The only part of the code that is shared with the cli right now is the new sd_webui_bayesian_merger.cli_args module, which exposes in a single location the defaults and choices of the command line arguments and of the user interface.

if cli is not used anymore, can we make the gui read params from config files?

I don't see why we wouldn't be able to. We may want to look into this indeed.

I like the idea of showing the UNET in the GUI, perhaps the viz could be updated while the optimisation runs too

I'll see what I can do, but as the gradio queue is disabled by the webui by default, it is not possible to register an event that is called periodically from python code. We may need to look into a custom javascript solution here.

[ljleb]

Does this look good enough for you? I can move stuff around a bit if you don't like it. I'll be honest, I took a bit of inspiration from the supermerger extension for the placement because I never really tried to put together a convenient user interface before.

[ljleb]

One thing I need to verify is, does skip_position_ids correspond to the value of shared.opts.CLIP_stop_at_last_layers - 1 (aka clip skip) in the webui? At the moment I just take the value from the webui and pass it to the script. Maybe we want it to be a ui parameter?

Another thing the user interface does is read the browser's window.location.href value, which basically corresponds to the url that appears in the browser bar at the top, and pass it as the API endpoint for the python backend. I am not aware of a better approach, I'd like to know if anyone has a better idea. Note that the endpoint url would not be needed if we sampled images directly from within the webui instead of going through the API.

[s1dlx]

Love it!

how difficult would be to have a menu to select the scorer model instead of typing the name?

Yes, that is the clip skip value.

[ljleb]

Thanks!

I was thinking of adding a file picker instead, for example in case you want to use a finetuned scorer or just any model that uses the structure of one of the supported types.

Alternatively, I can create a dropdown based on the selected folder perhaps? And then add a refresh button to fill the dropdown with the contents of the folder.

[s1dlx]

No strong opinion on this tbh

perhaps we should follow the drop down + refresh button that seems to be used a lot in webui

[ljleb]

I agree. I think the scorer directory could be a setting in the options tab.

We may want to add <a1111 sd webui root>/models/scorer to search paths by default on top of the path relative to the extension.

[s1dlx]

@ljleb what's the status on this?

I was thinking that the number of things we're adding to this repository is going to be too large to keep up with the UI.

An idea may be to let the user define all the parameters in the config files, while the GUI contains only results/images/plots

[ljleb]

An idea may be to let the user define all the parameters in the config files, while the GUI contains only results/images/plots

Yes that is what is implemented in the interface at the moment, (at least for the payloads) although I believe some of the options dont necessarily make sense because I have not used the extension enough to understand the purpose of everything. But yeah I think this is similar to how you need to configure bmaltais/kohya_ss before creating a lora.

I have a couple PRs open here and there in different repos, I need to actually finish the work and I will have to prioritize. I am also a little busy at the moment so if you or someone else want to look into finishing this or take inspiration from the ui code that is already there feel free to do it.

Except for how the gradio components are declared (i.e. using metaclasses), i think the code is structured in a way that allows easier maintainability, by separating the components in 1) definition, (metaclass fields) 2) visual organisation, (rearrange_components) and 3) dynamic behavior with user interraction. (connect_events)

[ljleb]

Oops, did not intend to close. I misclicked the button while typing my response.

[ljleb]

So for payloads, it makes sense to ask for a path and have users manually edit the payloads I think. For all other config parameters however, I think it makes more sense to have them in the gradio ui.

This means bypassing the hydra config files to launch the optimizer, and also having to maintain the ui parameters whenever new configuration options are added or old ones are removed. We can load the configuration from /conf for default ui values.

So there is a tradeoff of code maintenance vs more convenience on the usage side, if we do this instead of just asking for the hydra config path.

[ljleb]

For now I'll remove the ui components and use the config directly. We can look into an easier way to maintain the parameters later, and provide the ui parameters corresponding to the config whenever it is easier to keep everything up to date.