[Feature]: Option to do post-process upscale before extensions/scripts

[MysticDaedra]

Feature description

This used to be the order of operations in SDNext until a recent update, not sure which one to be honest. Anyways, use-case examples:

• Limited VRAM for initial generation/hires, upscale before extensions or scripts so that more resolution and data is available to them
• upscale individual images in an XYZ grid (current implementation would upscale the entire finished grid!)

Honestly, the flexibility to upscale before extensions probably could provide many more use-cases or workflows than the two I posted above. Below is my desired and previously most-commonly used workflow:

1. Initial generation
2. 1.5-1.7 (depending on model and resolution) upscale + hires fix
3. Upscale 2x or 4x
4. adetailer for additional detail on characters or specific scene elements, greatly benefits from higher resolution initial images
5. Last upscale (if it's a seed or prompt I really like) usually x2. Or XYZ grid would go here potentially.

Version Platform Description

Current dev branch 59cd08f

[vladmandic]

lets split this logically - anything to do with xyz is does not belong here, we're talking about a completely different story.
and if that is removed, i'm really not sure what you'd expect as your example workflow is
generate-> upscale -> hires -> upscale -> detailer -> upscale

right now you can do:
generate -> upscale -> hires -> detailer -> upscale

and i really don't see the benefits of adding one more upscale before detailer, not to mention there are no ui elements for it and it would make everything extremely messy to configure.

also, look at control tab workflows - it has much more flexible scaling as it does have before/after concept.

so lets re-think this - what is the exact desired minimum workflow?

[vladmandic]

@MysticDaedra your thoughts on the previous post?

[MysticDaedra]

@MysticDaedra your thoughts on the previous post?

Sorry, I didn't notice that you had replied, my bad.

I don't use detailer myself, I used adetailer, I find it much more flexible, although I respect your reasoning for keeping detailer more simple. That's a topic for a different time however (I've also expressed my issues with the implementation/planned scope of detailer on discord previously).

My specific use-case benefits greatly from additional data being available in the image prior to inference (img2img), as adetailer or any img2img process is higher quality and provides greater details when the source image is a higher resolution. Due to how adetailer works, relatively high resolution images can be "fed" to it without causing OOM on lower VRAM GPUs, as the actual inpainting region resolution is relatively low (unless you get crazy I suppose).

This doesn't work with hires, because hires works on the entire image, so if you upscale by too much, you'll still get an OOM on lower VRAM (or even higher VRAM depending on how much you push the resolution) GPUs. Again, that is my specific use-case. However, the principle, in my opinion, holds true for other extensions or scripts that might be applied during the "post-process" part of image generation, anything after initial inference and hires.

I want to note a couple of things: first, this functionality existed prior to November. I, again, was rather surprised when I found that this functionality had effectively been removed in a recent update. Second, the principle behind changing the order of post-process processes already exists in settings, you can change when several different post-process functions are applied, such as upscaling, codeformer etc. Unless extensions and scripts are actually outside the order of operations, in which case... idk.

RE: the control tab, I need to revisit that and see if it has what I'm looking for, although I generally don't use it because there are tons of options (including controlnet itself lol) that I don't use for mostly VRAM-related reasons. With post-process you can choose to hide options within the system post-processing menu, but afaik you can't hide tabs and what-not in the control tab the same way.

To re-iterate, the desired functionality, as I can recall it working previously, is inference>upscale>hires>upscale>adetailer/extensions/scripts.

[vladmandic]

I want to note a couple of things: first, this functionality existed prior to November. I, again, was rather surprised when I found that this functionality had effectively been removed in a recent update.

I'm not sure I understand. the code cleanups may have moved things around, but no such explicit functionality ever existed.
But all this is kind of moot, start using control and detailer and then we can see how we can improve it end-to-end. I really cannot be looking at how to improve all-things-legacy.