-
+
**Image Diffusion implementation with advanced features**
@@ -8,15 +8,16 @@
[](https://discord.gg/VjvR2tabEX)
[](https://github.com/sponsors/vladmandic)
-[Wiki](https://github.com/vladmandic/automatic/wiki) | [Discord](https://discord.gg/VjvR2tabEX) | [Changelog](CHANGELOG.md)
+[Docs](https://vladmandic.github.io/sdnext-docs/) | [Wiki](https://github.com/vladmandic/automatic/wiki) | [Discord](https://discord.gg/VjvR2tabEX) | [Changelog](CHANGELOG.md)
## Table of contents
+- [Documentation](https://vladmandic.github.io/sdnext-docs/)
- [SD.Next Features](#sdnext-features)
-- [Model support](#model-support)
+- [Model support](#model-support) and [Specifications]()
- [Platform support](#platform-support)
- [Getting started](#getting-started)
@@ -25,7 +26,7 @@
All individual features are not listed here, instead check [ChangeLog](CHANGELOG.md) for full list of changes
- Multiple UIs!
▹ **Standard | Modern**
-- Multiple diffusion models!
+- Multiple [diffusion models](https://vladmandic.github.io/sdnext-docs/Model-Support/)!
- Built-in Control for Text, Image, Batch and video processing!
- Multiplatform!
▹ **Windows | Linux | MacOS | nVidia | AMD | IntelArc/IPEX | DirectML | OpenVINO | ONNX+Olive | ZLUDA**
@@ -34,9 +35,7 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
- Platform specific autodetection and tuning performed on install
- Optimized processing with latest `torch` developments with built-in support for `torch.compile`
and multiple compile backends: *Triton, ZLUDA, StableFast, DeepCache, OpenVINO, NNCF, IPEX, OneDiff*
-- Improved prompt parser
- Built-in queue management
-- Enterprise level logging and hardened API
- Built in installer with automatic updates and dependency management
- Mobile compatible
@@ -49,42 +48,13 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
-For screenshots and informations on other available themes, see [Themes Wiki](https://github.com/vladmandic/automatic/wiki/Themes)
+For screenshots and informations on other available themes, see [Themes](https://vladmandic.github.io/sdnext-docs/Themes/)
+
**Image Diffusion implementation with advanced features**
@@ -8,15 +8,16 @@
[](https://discord.gg/VjvR2tabEX)
[](https://github.com/sponsors/vladmandic)
-[Wiki](https://github.com/vladmandic/automatic/wiki) | [Discord](https://discord.gg/VjvR2tabEX) | [Changelog](CHANGELOG.md)
+[Docs](https://vladmandic.github.io/sdnext-docs/) | [Wiki](https://github.com/vladmandic/automatic/wiki) | [Discord](https://discord.gg/VjvR2tabEX) | [Changelog](CHANGELOG.md)
## Model support -Additional models will be added as they become available and there is public interest in them -See [models overview](https://github.com/vladmandic/automatic/wiki/Models) for details on each model, including their architecture, complexity and other info - -- [RunwayML Stable Diffusion](https://github.com/Stability-AI/stablediffusion/) 1.x and 2.x *(all variants)* -- [StabilityAI Stable Diffusion XL](https://github.com/Stability-AI/generative-models), [StabilityAI Stable Diffusion 3.0](https://stability.ai/news/stable-diffusion-3-medium) Medium, [StabilityAI Stable Diffusion 3.5](https://huggingface.co/stabilityai/stable-diffusion-3.5-large) Medium, Large, Large Turbo -- [StabilityAI Stable Video Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid) Base, XT 1.0, XT 1.1 -- [StabilityAI Stable Cascade](https://github.com/Stability-AI/StableCascade) *Full* and *Lite* -- [Black Forest Labs FLUX.1](https://blackforestlabs.ai/announcing-black-forest-labs/) Dev, Schnell -- [AuraFlow](https://huggingface.co/fal/AuraFlow) -- [AlphaVLLM Lumina-Next-SFT](https://huggingface.co/Alpha-VLLM/Lumina-Next-SFT-diffusers) -- [Playground AI](https://huggingface.co/playgroundai/playground-v2-256px-base) *v1, v2 256, v2 512, v2 1024 and latest v2.5* -- [Tencent HunyuanDiT](https://github.com/Tencent/HunyuanDiT) -- [OmniGen](https://arxiv.org/pdf/2409.11340) -- [Meissonic](https://github.com/viiika/Meissonic) -- [Kwai Kolors](https://huggingface.co/Kwai-Kolors/Kolors) -- [CogView 3+](https://huggingface.co/THUDM/CogView3-Plus-3B) -- [LCM: Latent Consistency Models](https://github.com/openai/consistency_models) -- [aMUSEd](https://huggingface.co/amused/amused-256) 256 and 512 -- [Segmind Vega](https://huggingface.co/segmind/Segmind-Vega), [Segmind SSD-1B](https://huggingface.co/segmind/SSD-1B), [Segmind SegMoE](https://github.com/segmind/segmoe) *SD and SD-XL*, [Segmind SD Distilled](https://huggingface.co/blog/sd_distillation) *(all variants)* -- [Kandinsky](https://github.com/ai-forever/Kandinsky-2) *2.1 and 2.2 and latest 3.0* -- [PixArt-α XL 2](https://github.com/PixArt-alpha/PixArt-alpha) *Medium and Large*, [PixArt-Σ](https://github.com/PixArt-alpha/PixArt-sigma) -- [Warp Wuerstchen](https://huggingface.co/blog/wuertschen) -- [Tsinghua UniDiffusion](https://github.com/thu-ml/unidiffuser) -- [DeepFloyd IF](https://github.com/deep-floyd/IF) *Medium and Large* -- [ModelScope T2V](https://huggingface.co/damo-vilab/text-to-video-ms-1.7b) -- [BLIP-Diffusion](https://dxli94.github.io/BLIP-Diffusion-website/) -- [KOALA 700M](https://github.com/youngwanLEE/sdxl-koala) -- [VGen](https://huggingface.co/ali-vilab/i2vgen-xl) -- [SDXS](https://github.com/IDKiro/sdxs) -- [Hyper-SD](https://huggingface.co/ByteDance/Hyper-SD) +SD.Next supports broad range of models: [supported models](https://vladmandic.github.io/sdnext-docs/Model-Support/) and [model specs](https://vladmandic.github.io/sdnext-docs/Models/) ## Platform support @@ -97,47 +67,29 @@ See [models overview](https://github.com/vladmandic/automatic/wiki/Models) for d - Any GPU or device compatible with **OpenVINO** libraries on both *Windows and Linux* - *Apple M1/M2* on *OSX* using built-in support in Torch with **MPS** optimizations - *ONNX/Olive* -- *AMD* GPUs on Windows using **ZLUDA** libraries +- *AMD* GPUs on Windows using **ZLUDA** libraries ## Getting started -- Get started with **SD.Next** by following the [installation instructions](https://github.com/vladmandic/automatic/wiki/Installation) -- For more details, check out [advanced installation](https://github.com/vladmandic/automatic/wiki/Advanced-Install) guide -- List and explanation of [command line arguments](https://github.com/vladmandic/automatic/wiki/CLI-Arguments) +- Get started with **SD.Next** by following the [installation instructions](https://vladmandic.github.io/sdnext-docs/Installation/) +- For more details, check out [advanced installation](https://vladmandic.github.io/sdnext-docs/Advanced-Install/) guide +- List and explanation of [command line arguments](https://vladmandic.github.io/sdnext-docs/CLI-Arguments/) - Install walkthrough [video](https://www.youtube.com/watch?v=nWTnTyFTuAs) > [!TIP] > And for platform specific information, check out -> [WSL](https://github.com/vladmandic/automatic/wiki/WSL) | [Intel Arc](https://github.com/vladmandic/automatic/wiki/Intel-ARC) | [DirectML](https://github.com/vladmandic/automatic/wiki/DirectML) | [OpenVINO](https://github.com/vladmandic/automatic/wiki/OpenVINO) | [ONNX & Olive](https://github.com/vladmandic/automatic/wiki/ONNX-Runtime) | [ZLUDA](https://github.com/vladmandic/automatic/wiki/ZLUDA) | [AMD ROCm](https://github.com/vladmandic/automatic/wiki/AMD-ROCm) | [MacOS](https://github.com/vladmandic/automatic/wiki/MacOS-Python.md) | [nVidia](https://github.com/vladmandic/automatic/wiki/nVidia) +> [WSL](https://vladmandic.github.io/sdnext-docs/WSL/) | [Intel Arc](https://vladmandic.github.io/sdnext-docs/Intel-ARC/) | [DirectML](https://vladmandic.github.io/sdnext-docs/DirectML/) | [OpenVINO](https://vladmandic.github.io/sdnext-docs/OpenVINO/) | [ONNX & Olive](https://vladmandic.github.io/sdnext-docs/ONNX-Runtime/) | [ZLUDA](https://vladmandic.github.io/sdnext-docs/ZLUDA/) | [AMD ROCm](https://vladmandic.github.io/sdnext-docs/AMD-ROCm/) | [MacOS](https://vladmandic.github.io/sdnext-docs/MacOS-Python/) | [nVidia](https://vladmandic.github.io/sdnext-docs/nVidia/) | [Docker](https://vladmandic.github.io/sdnext-docs/Docker/) > [!WARNING] -> If you run into issues, check out [troubleshooting](https://github.com/vladmandic/automatic/wiki/Troubleshooting) and [debugging](https://github.com/vladmandic/automatic/wiki/Debug) guides +> If you run into issues, check out [troubleshooting](https://vladmandic.github.io/sdnext-docs/Troubleshooting/) and [debugging](https://vladmandic.github.io/sdnext-docs/Debug/) guides > [!TIP] -> All command line options can also be set via env variable +> All command line options can also be set via env variable > For example `--debug` is same as `set SD_DEBUG=true` -## Backend support - -**SD.Next** supports two main backends: *Diffusers* and *Original*: - -- **Diffusers**: Based on new [Huggingface Diffusers](https://huggingface.co/docs/diffusers/index) implementation - Supports *all* models listed below - This backend is set as default for new installations -- **Original**: Based on [LDM](https://github.com/Stability-AI/stablediffusion) reference implementation and significantly expanded on by [A1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui) - This backend and is fully compatible with most existing functionality and extensions written for *A1111 SDWebUI* - Supports **SD 1.x** and **SD 2.x** models - All other model types such as *SD-XL, LCM, Stable Cascade, PixArt, Playground, Segmind, Kandinsky, etc.* require backend **Diffusers** - -### Collab - -- We'd love to have additional maintainers (with comes with full repo rights). If you're interested, ping us! -- In addition to general cross-platform code, desire is to have a lead for each of the main platforms -This should be fully cross-platform, but we'd really love to have additional contributors and/or maintainers to join and help lead the efforts on different platforms - ### Credits -- Main credit goes to [Automatic1111 WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) for original codebase +- Main credit goes to [Automatic1111 WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) for the original codebase - Additional credits are listed in [Credits](https://github.com/AUTOMATIC1111/stable-diffusion-webui/#credits) - Licenses for modules are listed in [Licenses](html/licenses.html) @@ -154,8 +106,8 @@ This should be fully cross-platform, but we'd really love to have additional con ### Docs -If you're unsure how to use a feature, best place to start is [Wiki](https://github.com/vladmandic/automatic/wiki) and if its not there, -check [ChangeLog](CHANGELOG.md) for when feature was first introduced as it will always have a short note on how to use it +If you're unsure how to use a feature, best place to start is [Docs](https://vladmandic.github.io/sdnext-docs/) and if its not there, +check [ChangeLog](https://vladmandic.github.io/sdnext-docs/CHANGELOG/) for when feature was first introduced as it will always have a short note on how to use it ### Sponsors diff --git a/TODO.md b/TODO.md index 973e062dc..6d89c838f 100644 --- a/TODO.md +++ b/TODO.md @@ -2,21 +2,30 @@ Main ToDo list can be found at [GitHub projects](https://github.com/users/vladmandic/projects) -## Future Candidates +## Pending -- SD35 IPAdapter:
" - vram_html += f"GPU active {max(vram['active_peak'], vram['reserved_peak'])} MB reserved {vram['reserved']} | used {vram['used']} MB free {vram['free']} MB total {vram['total']} MB" - vram_html += f" | retries {vram['retries']} oom {vram['oom']}" if vram.get('retries', 0) > 0 or vram.get('oom', 0) > 0 else '' - vram_html += "
" + peak = max(vram['active_peak'], vram['reserved_peak'], vram['used']) + used = round(100.0 * peak / vram['total']) if vram['total'] > 0 else 0 + if used > 0: + gpu += f"| GPU {peak} MB {used}%" + gpu += f" | retries {vram['retries']} oom {vram['oom']}" if vram.get('retries', 0) > 0 or vram.get('oom', 0) > 0 else '' + ram = shared.ram_stats() + if ram['used'] > 0: + cpu += f"| RAM {ram['used']} GB {round(100.0 * ram['used'] / ram['total'])}%" if isinstance(res, list): - res[-1] += f"Time: {elapsed_text}
{vram_html}Time: {elapsed_text} | {summary} {gpu} {cpu}
Ready {image_txt}
' + html_txt = f'Ready {image_txt}
' if image_txt != '' else '' if len(info_txt) > 0: html_txt = html_txt + infotext_to_html(info_txt[0]) if is_generator: yield (output_images, blended_image, html_txt, output_filename) - else: - return (output_images, blended_image, html_txt, output_filename) + return (output_images, blended_image, html_txt, output_filename) diff --git a/modules/control/tile.py b/modules/control/tile.py new file mode 100644 index 000000000..de9df1131 --- /dev/null +++ b/modules/control/tile.py @@ -0,0 +1,73 @@ +import time +from PIL import Image +from modules import shared, processing, images, sd_models + + +def get_tile(image: Image.Image, x: int, y: int, sx: int, sy: int) -> Image.Image: + return image.crop(( + (x + 0) * image.width // sx, + (y + 0) * image.height // sy, + (x + 1) * image.width // sx, + (y + 1) * image.height // sy + )) + + +def set_tile(image: Image.Image, x: int, y: int, tiled: Image.Image): + image.paste(tiled, (x * tiled.width, y * tiled.height)) + return image + + +def run_tiling(p: processing.StableDiffusionProcessing, input_image: Image.Image) -> processing.Processed: + t0 = time.time() + # prepare images + sx, sy = p.control_tile.split('x') + sx = int(sx) + sy = int(sy) + if sx <= 0 or sy <= 0: + raise ValueError('Control Tile: invalid tile size') + control_image = p.task_args.get('control_image', None) or p.task_args.get('image', None) + control_upscaled = None + if isinstance(control_image, list) and len(control_image) > 0: + w, h = 8 * int(sx * control_image[0].width) // 8, 8 * int(sy * control_image[0].height) // 8 + control_upscaled = images.resize_image(resize_mode=1 if sx==sy else 5, im=control_image[0], width=w, height=h, context='add with forward') + init_image = p.override or input_image + init_upscaled = None + if init_image is not None: + w, h = 8 * int(sx * init_image.width) // 8, 8 * int(sy * init_image.height) // 8 + init_upscaled = images.resize_image(resize_mode=1 if sx==sy else 5, im=init_image, width=w, height=h, context='add with forward') + t1 = time.time() + shared.log.debug(f'Control Tile: scale={sx}x{sy} resize={"fixed" if sx==sy else "context"} control={control_upscaled} init={init_upscaled} time={t1-t0:.3f}') + + # stop processing from restoring pipeline on each iteration + orig_restore_pipeline = getattr(shared.sd_model, 'restore_pipeline', None) + shared.sd_model.restore_pipeline = None + + # run tiling + for x in range(sx): + for y in range(sy): + shared.log.info(f'Control Tile: tile={x+1}-{sx}/{y+1}-{sy} target={control_upscaled}') + shared.sd_model = sd_models.set_diffuser_pipe(shared.sd_model, sd_models.DiffusersTaskType.IMAGE_2_IMAGE) + p.init_images = None + p.task_args['control_mode'] = p.control_mode + p.task_args['strength'] = p.denoising_strength + if init_upscaled is not None: + p.task_args['image'] = [get_tile(init_upscaled, x, y, sx, sy)] + if control_upscaled is not None: + p.task_args['control_image'] = [get_tile(control_upscaled, x, y, sx, sy)] + processed: processing.Processed = processing.process_images(p) # run actual pipeline + if processed is None or len(processed.images) == 0: + continue + control_upscaled = set_tile(control_upscaled, x, y, processed.images[0]) + + # post-process + p.width = control_upscaled.width + p.height = control_upscaled.height + processed.images = [control_upscaled] + processed.info = processed.infotext(p, 0) + processed.infotexts = [processed.info] + shared.sd_model.restore_pipeline = orig_restore_pipeline + if hasattr(shared.sd_model, 'restore_pipeline') and shared.sd_model.restore_pipeline is not None: + shared.sd_model.restore_pipeline() + t2 = time.time() + shared.log.debug(f'Control Tile: image={control_upscaled} time={t2-t0:.3f}') + return processed diff --git a/modules/control/unit.py b/modules/control/unit.py index 7dc5528a6..eeb729740 100644 --- a/modules/control/unit.py +++ b/modules/control/unit.py @@ -16,6 +16,22 @@ class Unit(): # mashup of gradio controls and mapping to actual implementation classes + def update_choices(self, model_id=None): + name = model_id or self.model_name + if name == 'InstantX Union': + self.choices = ['canny', 'tile', 'depth', 'blur', 'pose', 'gray', 'lq'] + elif name == 'Shakker-Labs Union': + self.choices = ['canny', 'tile', 'depth', 'blur', 'pose', 'gray', 'lq'] + elif name == 'Xinsir Union XL': + self.choices = ['openpose', 'depth', 'scribble', 'canny', 'normal'] + elif name == 'Xinsir ProMax XL': + self.choices = ['openpose', 'depth', 'scribble', 'canny', 'normal', 'segment', 'tile', 'repaint'] + else: + self.choices = ['default'] + + def __str__(self): + return f'Unit: type={self.type} enabled={self.enabled} strength={self.strength} start={self.start} end={self.end} mode={self.mode} tile={self.tile}' + def __init__(self, # values index: int = None, @@ -38,6 +54,7 @@ def __init__(self, control_start = None, control_end = None, control_mode = None, + control_tile = None, result_txt = None, extra_controls: list = [], ): @@ -70,6 +87,10 @@ def __init__(self, self.fidelity = 0.5 self.query_weight = 1.0 self.adain_weight = 1.0 + # control mode + self.choices = ['default'] + # control tile + self.tile = '1x1' def reset(): if self.process is not None: @@ -92,10 +113,16 @@ def control_change(start, end): self.end = max(start, end) def control_mode_change(mode): - self.mode = mode - 1 if mode > 0 else None + self.mode = self.choices.index(mode) if mode is not None and mode in self.choices else 0 + + def control_tile_change(tile): + self.tile = tile - def control_mode_show(model_id): - return gr.update(visible='union' in model_id.lower()) + def control_choices(model_id): + self.update_choices(model_id) + mode_visible = 'union' in model_id.lower() or 'promax' in model_id.lower() + tile_visible = 'union' in model_id.lower() or 'promax' in model_id.lower() or 'tile' in model_id.lower() + return [gr.update(visible=mode_visible, choices=self.choices), gr.update(visible=tile_visible)] def adapter_extra(c1): self.factor = c1 @@ -172,7 +199,7 @@ def set_image(image): else: self.controls.append(model_id) model_id.change(fn=self.controlnet.load, inputs=[model_id], outputs=[result_txt], show_progress=True) - model_id.change(fn=control_mode_show, inputs=[model_id], outputs=[control_mode], show_progress=False) + model_id.change(fn=control_choices, inputs=[model_id], outputs=[control_mode, control_tile], show_progress=False) if extra_controls is not None and len(extra_controls) > 0: extra_controls[0].change(fn=controlnet_extra, inputs=extra_controls) elif self.type == 'xs': @@ -231,3 +258,6 @@ def set_image(image): if control_mode is not None: self.controls.append(control_mode) control_mode.change(fn=control_mode_change, inputs=[control_mode]) + if control_tile is not None: + self.controls.append(control_tile) + control_tile.change(fn=control_tile_change, inputs=[control_tile]) diff --git a/modules/control/units/controlnet.py b/modules/control/units/controlnet.py index 20b99412a..c887aca8f 100644 --- a/modules/control/units/controlnet.py +++ b/modules/control/units/controlnet.py @@ -5,6 +5,7 @@ from modules.control.units import detect from modules.shared import log, opts, listdir from modules import errors, sd_models, devices, model_quant +from modules.processing import StableDiffusionProcessingControl what = 'ControlNet' @@ -51,17 +52,20 @@ 'Depth Mid XL': 'diffusers/controlnet-depth-sdxl-1.0-mid', 'OpenPose XL': 'thibaud/controlnet-openpose-sdxl-1.0/bin', 'Xinsir Union XL': 'xinsir/controlnet-union-sdxl-1.0', + 'Xinsir ProMax XL': 'brad-twinkl/controlnet-union-sdxl-1.0-promax', 'Xinsir OpenPose XL': 'xinsir/controlnet-openpose-sdxl-1.0', 'Xinsir Canny XL': 'xinsir/controlnet-canny-sdxl-1.0', 'Xinsir Depth XL': 'xinsir/controlnet-depth-sdxl-1.0', 'Xinsir Scribble XL': 'xinsir/controlnet-scribble-sdxl-1.0', 'Xinsir Anime Painter XL': 'xinsir/anime-painter', + 'Xinsir Tile XL': 'xinsir/controlnet-tile-sdxl-1.0', 'NoobAI Canny XL': 'Eugeoter/noob-sdxl-controlnet-canny', 'NoobAI Lineart Anime XL': 'Eugeoter/noob-sdxl-controlnet-lineart_anime', 'NoobAI Depth XL': 'Eugeoter/noob-sdxl-controlnet-depth', 'NoobAI Normal XL': 'Eugeoter/noob-sdxl-controlnet-normal', 'NoobAI SoftEdge XL': 'Eugeoter/noob-sdxl-controlnet-softedge_hed', 'NoobAI OpenPose XL': 'einar77/noob-openpose', + 'TTPlanet Tile Realistic XL': 'Yakonrus/SDXL_Controlnet_Tile_Realistic_v2', # 'StabilityAI Canny R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-canny-rank128.safetensors', # 'StabilityAI Depth R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-depth-rank128.safetensors', # 'StabilityAI Recolor R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-recolor-rank128.safetensors', @@ -75,6 +79,8 @@ "InstantX Union": 'InstantX/FLUX.1-dev-Controlnet-Union', "InstantX Canny": 'InstantX/FLUX.1-dev-Controlnet-Canny', "JasperAI Depth": 'jasperai/Flux.1-dev-Controlnet-Depth', + "BlackForrestLabs Canny LoRA": '/huggingface.co/black-forest-labs/FLUX.1-Canny-dev-lora/flux1-canny-dev-lora.safetensors', + "BlackForrestLabs Depth LoRA": '/huggingface.co/black-forest-labs/FLUX.1-Depth-dev-lora/flux1-depth-dev-lora.safetensors', "JasperAI Surface Normals": 'jasperai/Flux.1-dev-Controlnet-Surface-Normals', "JasperAI Upscaler": 'jasperai/Flux.1-dev-Controlnet-Upscaler', "Shakker-Labs Union": 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro', @@ -85,6 +91,9 @@ "XLabs-AI HED": 'XLabs-AI/flux-controlnet-hed-diffusers' } predefined_sd3 = { + "StabilityAI Canny": 'diffusers-internal-dev/sd35-controlnet-canny-8b', + "StabilityAI Depth": 'diffusers-internal-dev/sd35-controlnet-depth-8b', + "StabilityAI Blur": 'diffusers-internal-dev/sd35-controlnet-blur-8b', "InstantX Canny": 'InstantX/SD3-Controlnet-Canny', "InstantX Pose": 'InstantX/SD3-Controlnet-Pose', "InstantX Depth": 'InstantX/SD3-Controlnet-Depth', @@ -92,6 +101,14 @@ "Alimama Inpainting": 'alimama-creative/SD3-Controlnet-Inpainting', "Alimama SoftEdge": 'alimama-creative/SD3-Controlnet-Softedge', } +variants = { + 'NoobAI Canny XL': 'fp16', + 'NoobAI Lineart Anime XL': 'fp16', + 'NoobAI Depth XL': 'fp16', + 'NoobAI Normal XL': 'fp16', + 'NoobAI SoftEdge XL': 'fp16', + 'TTPlanet Tile Realistic XL': 'fp16', +} models = {} all_models = {} all_models.update(predefined_sd15) @@ -159,26 +176,35 @@ def reset(self): self.model = None self.model_id = None - def get_class(self): - import modules.shared - if modules.shared.sd_model_type == 'sd': + def get_class(self, model_id:str=''): + from modules import shared + if shared.sd_model_type == 'none': + _load = shared.sd_model # trigger a load + if shared.sd_model_type == 'sd': from diffusers import ControlNetModel as cls # pylint: disable=reimported config = 'lllyasviel/control_v11p_sd15_canny' - elif modules.shared.sd_model_type == 'sdxl': - from diffusers import ControlNetModel as cls # pylint: disable=reimported # sdxl shares same model class - config = 'Eugeoter/noob-sdxl-controlnet-canny' - elif modules.shared.sd_model_type == 'f1': + elif shared.sd_model_type == 'sdxl': + if 'union' in model_id.lower(): + from diffusers import ControlNetUnionModel as cls + config = 'xinsir/controlnet-union-sdxl-1.0' + elif 'promax' in model_id.lower(): + from diffusers import ControlNetUnionModel as cls + config = 'brad-twinkl/controlnet-union-sdxl-1.0-promax' + else: + from diffusers import ControlNetModel as cls # pylint: disable=reimported # sdxl shares same model class + config = 'Eugeoter/noob-sdxl-controlnet-canny' + elif shared.sd_model_type == 'f1': from diffusers import FluxControlNetModel as cls config = 'InstantX/FLUX.1-dev-Controlnet-Union' - elif modules.shared.sd_model_type == 'sd3': + elif shared.sd_model_type == 'sd3': from diffusers import SD3ControlNetModel as cls config = 'InstantX/SD3-Controlnet-Canny' else: - log.error(f'Control {what}: type={modules.shared.sd_model_type} unsupported model') + log.error(f'Control {what}: type={shared.sd_model_type} unsupported model') return None, None return cls, config - def load_safetensors(self, model_path): + def load_safetensors(self, model_id, model_path): name = os.path.splitext(model_path)[0] config_path = None if not os.path.exists(model_path): @@ -203,7 +229,7 @@ def load_safetensors(self, model_path): config_path = f'{name}.json' if config_path is not None: self.load_config['original_config_file '] = config_path - cls, config = self.get_class() + cls, config = self.get_class(model_id) if cls is None: log.error(f'Control {what} model load failed: unknown base model') else: @@ -225,23 +251,26 @@ def load(self, model_id: str = None, force: bool = True) -> str: if model_path is None: log.error(f'Control {what} model load failed: id="{model_id}" error=unknown model id') return + if 'lora' in model_id.lower(): + self.model = model_path + return if model_id == self.model_id and not force: log.debug(f'Control {what} model: id="{model_id}" path="{model_path}" already loaded') return log.debug(f'Control {what} model loading: id="{model_id}" path="{model_path}"') + cls, _config = self.get_class(model_id) if model_path.endswith('.safetensors'): - self.load_safetensors(model_path) + self.load_safetensors(model_id, model_path) else: kwargs = {} if '/bin' in model_path: model_path = model_path.replace('/bin', '') self.load_config['use_safetensors'] = False - cls, _config = self.get_class() if cls is None: log.error(f'Control {what} model load failed: id="{model_id}" unknown base model') return - if 'Eugeoter' in model_path: - kwargs['variant'] = 'fp16' + if variants.get(model_id, None) is not None: + kwargs['variant'] = variants[model_id] self.model = cls.from_pretrained(model_path, **self.load_config, **kwargs) if self.model is None: return @@ -268,7 +297,7 @@ def load(self, model_id: str = None, force: bool = True) -> str: self.model.to(self.device) t1 = time.time() self.model_id = model_id - log.debug(f'Control {what} model loaded: id="{model_id}" path="{model_path}" time={t1-t0:.2f}') + log.debug(f'Control {what} model loaded: id="{model_id}" path="{model_path}" cls={cls.__name__} time={t1-t0:.2f}') return f'{what} loaded model: {model_id}' except Exception as e: log.error(f'Control {what} model load failed: id="{model_id}" error={e}') @@ -281,16 +310,27 @@ def __init__(self, controlnet: Union[ControlNetModel, list[ControlNetModel]], pipeline: Union[StableDiffusionXLPipeline, StableDiffusionPipeline, FluxPipeline, StableDiffusion3Pipeline], dtype = None, + p: StableDiffusionProcessingControl = None, # pylint: disable=unused-argument ): t0 = time.time() self.orig_pipeline = pipeline self.pipeline = None + + controlnets = controlnet if isinstance(controlnet, list) else [controlnet] + loras = [cn for cn in controlnets if isinstance(cn, str)] + controlnets = [cn for cn in controlnets if not isinstance(cn, str)] + if pipeline is None: log.error('Control model pipeline: model not loaded') return - elif detect.is_sdxl(pipeline): - from diffusers import StableDiffusionXLControlNetPipeline - self.pipeline = StableDiffusionXLControlNetPipeline( + elif detect.is_sdxl(pipeline) and len(controlnets) > 0: + from diffusers import StableDiffusionXLControlNetPipeline, StableDiffusionXLControlNetUnionPipeline + if controlnet.__class__.__name__ == 'ControlNetUnionModel': + cls = StableDiffusionXLControlNetUnionPipeline + controlnets = controlnets[0] # using only first one + else: + cls = StableDiffusionXLControlNetPipeline + self.pipeline = cls( vae=pipeline.vae, text_encoder=pipeline.text_encoder, text_encoder_2=pipeline.text_encoder_2, @@ -299,9 +339,9 @@ def __init__(self, unet=pipeline.unet, scheduler=pipeline.scheduler, feature_extractor=getattr(pipeline, 'feature_extractor', None), - controlnet=controlnet, # can be a list + controlnet=controlnets, # can be a list ) - elif detect.is_sd15(pipeline): + elif detect.is_sd15(pipeline) and len(controlnets) > 0: from diffusers import StableDiffusionControlNetPipeline self.pipeline = StableDiffusionControlNetPipeline( vae=pipeline.vae, @@ -312,10 +352,10 @@ def __init__(self, feature_extractor=getattr(pipeline, 'feature_extractor', None), requires_safety_checker=False, safety_checker=None, - controlnet=controlnet, # can be a list + controlnet=controlnets, # can be a list ) sd_models.move_model(self.pipeline, pipeline.device) - elif detect.is_f1(pipeline): + elif detect.is_f1(pipeline) and len(controlnets) > 0: from diffusers import FluxControlNetPipeline self.pipeline = FluxControlNetPipeline( vae=pipeline.vae.to(devices.device), @@ -325,9 +365,9 @@ def __init__(self, tokenizer_2=pipeline.tokenizer_2, transformer=pipeline.transformer, scheduler=pipeline.scheduler, - controlnet=controlnet, # can be a list + controlnet=controlnets, # can be a list ) - elif detect.is_sd3(pipeline): + elif detect.is_sd3(pipeline) and len(controlnets) > 0: from diffusers import StableDiffusion3ControlNetPipeline self.pipeline = StableDiffusion3ControlNetPipeline( vae=pipeline.vae, @@ -339,8 +379,18 @@ def __init__(self, tokenizer_3=pipeline.tokenizer_3, transformer=pipeline.transformer, scheduler=pipeline.scheduler, - controlnet=controlnet, # can be a list + controlnet=controlnets, # can be a list ) + elif len(loras) > 0: + self.pipeline = pipeline + for lora in loras: + log.debug(f'Control {what} pipeline: lora="{lora}"') + lora = lora.replace('/huggingface.co/', '') + self.pipeline.load_lora_weights(lora) + """ + if p is not None: + p.prompt += f'Model Offloading
", "", gr.HTML), + "diffusers_move_base": OptionInfo(False, "Move base model to CPU when using refiner", gr.Checkbox, {"visible": False }), + "diffusers_move_unet": OptionInfo(False, "Move base model to CPU when using VAE", gr.Checkbox, {"visible": False }), + "diffusers_move_refiner": OptionInfo(False, "Move refiner model to CPU when not in use", gr.Checkbox, {"visible": False }), + "diffusers_extract_ema": OptionInfo(False, "Use model EMA weights when possible", gr.Checkbox, {"visible": False }), + "diffusers_offload_mode": OptionInfo(startup_offload_mode, "Model offload mode", gr.Radio, {"choices": ['none', 'balanced', 'model', 'sequential']}), + "diffusers_offload_min_gpu_memory": OptionInfo(0.25, "Balanced offload GPU low watermark", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01 }), + "diffusers_offload_max_gpu_memory": OptionInfo(0.70, "Balanced offload GPU high watermark", gr.Slider, {"minimum": 0.1, "maximum": 1, "step": 0.01 }), + "diffusers_offload_max_cpu_memory": OptionInfo(0.90, "Balanced offload CPU high watermark", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01, "visible": False }), + + "advanced_sep": OptionInfo("Advanced Options
", "", gr.HTML), "sd_checkpoint_autoload": OptionInfo(True, "Model autoload on start"), "sd_checkpoint_autodownload": OptionInfo(True, "Model auto-download on demand"), - "sd_textencoder_cache": OptionInfo(True, "Cache text encoder results", gr.Checkbox, {"visible": False}), - "sd_textencoder_cache_size": OptionInfo(4, "Text encoder results LRU cache size", gr.Slider, {"minimum": 0, "maximum": 10, "step": 1}), - "stream_load": OptionInfo(False, "Load models using stream loading method", gr.Checkbox, {"visible": not native }), + "stream_load": OptionInfo(False, "Model load using streams", gr.Checkbox), + "diffusers_eval": OptionInfo(True, "Force model eval", gr.Checkbox, {"visible": False }), + "diffusers_to_gpu": OptionInfo(False, "Load model directly to GPU"), + "disable_accelerate": OptionInfo(False, "Disable accelerate", gr.Checkbox, {"visible": False }), + "sd_model_dict": OptionInfo('None', "Use separate base dict", gr.Dropdown, lambda: {"choices": ['None'] + list_checkpoint_titles()}, refresh=refresh_checkpoints), + "sd_checkpoint_cache": OptionInfo(0, "Cached models", gr.Slider, {"minimum": 0, "maximum": 10, "step": 1, "visible": not native }), +})) + +options_templates.update(options_section(('vae_encoder', "Variable Auto Encoder"), { + "sd_vae": OptionInfo("Automatic", "VAE model", gr.Dropdown, lambda: {"choices": shared_items.sd_vae_items()}, refresh=shared_items.refresh_vae_list), + "diffusers_vae_upcast": OptionInfo("default", "VAE upcasting", gr.Radio, {"choices": ['default', 'true', 'false']}), + "no_half_vae": OptionInfo(False if not cmd_opts.use_openvino else True, "Full precision (--no-half-vae)"), + "diffusers_vae_slicing": OptionInfo(True, "VAE slicing", gr.Checkbox, {"visible": native}), + "diffusers_vae_tiling": OptionInfo(cmd_opts.lowvram or cmd_opts.medvram, "VAE tiling", gr.Checkbox, {"visible": native}), + "sd_vae_sliced_encode": OptionInfo(False, "VAE sliced encode", gr.Checkbox, {"visible": not native}), + "nan_skip": OptionInfo(False, "Skip Generation if NaN found in latents", gr.Checkbox), + "rollback_vae": OptionInfo(False, "Attempt VAE roll back for NaN values"), +})) + +options_templates.update(options_section(('text_encoder', "Text Encoder"), { + "sd_text_encoder": OptionInfo('None', "Text encoder model", gr.Dropdown, lambda: {"choices": shared_items.sd_te_items()}, refresh=shared_items.refresh_te_list), + "prompt_attention": OptionInfo("native", "Prompt attention parser", gr.Radio, {"choices": ["native", "compel", "xhinker", "a1111", "fixed"] }), "prompt_mean_norm": OptionInfo(False, "Prompt attention normalization", gr.Checkbox), + "sd_textencoder_cache": OptionInfo(True, "Cache text encoder results", gr.Checkbox, {"visible": False}), + "sd_textencoder_cache_size": OptionInfo(4, "Text encoder cache size", gr.Slider, {"minimum": 0, "maximum": 16, "step": 1}), "comma_padding_backtrack": OptionInfo(20, "Prompt padding", gr.Slider, {"minimum": 0, "maximum": 74, "step": 1, "visible": not native }), - "prompt_attention": OptionInfo("native", "Prompt attention parser", gr.Radio, {"choices": ["native", "compel", "xhinker", "a1111", "fixed"] }), - "latent_history": OptionInfo(16, "Latent history size", gr.Slider, {"minimum": 1, "maximum": 100, "step": 1}), - "sd_checkpoint_cache": OptionInfo(0, "Cached models", gr.Slider, {"minimum": 0, "maximum": 10, "step": 1, "visible": not native }), + "diffusers_zeros_prompt_pad": OptionInfo(False, "Use zeros for prompt padding", gr.Checkbox), + "diffusers_pooled": OptionInfo("default", "Diffusers SDXL pooled embeds", gr.Radio, {"choices": ['default', 'weighted']}), })) options_templates.update(options_section(('cuda', "Compute Settings"), { - "math_sep": OptionInfo("Execution precision
", "", gr.HTML), + "math_sep": OptionInfo("Execution Precision
", "", gr.HTML), "precision": OptionInfo("Autocast", "Precision type", gr.Radio, {"choices": ["Autocast", "Full"]}), "cuda_dtype": OptionInfo("Auto", "Device precision type", gr.Radio, {"choices": ["Auto", "FP32", "FP16", "BF16"]}), + "no_half": OptionInfo(False if not cmd_opts.use_openvino else True, "Full precision (--no-half)", None, None, None), + "upcast_sampling": OptionInfo(False if sys.platform != "darwin" else True, "Upcast sampling", gr.Checkbox, {"visible": not native}), + "upcast_attn": OptionInfo(False, "Upcast attention layer", gr.Checkbox, {"visible": not native}), + "cuda_cast_unet": OptionInfo(False, "Fixed UNet precision", gr.Checkbox, {"visible": not native}), - "model_sep": OptionInfo("Model options
", "", gr.HTML), - "no_half": OptionInfo(False if not cmd_opts.use_openvino else True, "Full precision for model (--no-half)", None, None, None), - "no_half_vae": OptionInfo(False if not cmd_opts.use_openvino else True, "Full precision for VAE (--no-half-vae)"), - "upcast_sampling": OptionInfo(False if sys.platform != "darwin" else True, "Upcast sampling"), - "upcast_attn": OptionInfo(False, "Upcast attention layer"), - "cuda_cast_unet": OptionInfo(False, "Fixed UNet precision"), - "nan_skip": OptionInfo(False, "Skip Generation if NaN found in latents", gr.Checkbox), - "rollback_vae": OptionInfo(False, "Attempt VAE roll back for NaN values"), + "generator_sep": OptionInfo("Noise Options
", "", gr.HTML), + "diffusers_generator_device": OptionInfo("GPU", "Generator device", gr.Radio, {"choices": ["GPU", "CPU", "Unset"]}), "cross_attention_sep": OptionInfo("Cross Attention
", "", gr.HTML), - "cross_attention_optimization": OptionInfo(startup_cross_attention, "Attention optimization method", gr.Radio, lambda: {"choices": shared_items.list_crossattention(native) }), - "sdp_options": OptionInfo(startup_sdp_options, "SDP options", gr.CheckboxGroup, {"choices": ['Flash attention', 'Memory attention', 'Math attention', 'Dynamic attention', 'Sage attention'] }), + "cross_attention_optimization": OptionInfo(startup_cross_attention, "Attention optimization method", gr.Radio, lambda: {"choices": shared_items.list_crossattention(native)}), + "sdp_options": OptionInfo(startup_sdp_options, "SDP options", gr.CheckboxGroup, {"choices": ['Flash attention', 'Memory attention', 'Math attention', 'Dynamic attention', 'Sage attention'], "visible": native}), "xformers_options": OptionInfo(['Flash attention'], "xFormers options", gr.CheckboxGroup, {"choices": ['Flash attention'] }), "dynamic_attention_slice_rate": OptionInfo(4, "Dynamic Attention slicing rate in GB", gr.Slider, {"minimum": 0.1, "maximum": gpu_memory, "step": 0.1, "visible": native}), "sub_quad_sep": OptionInfo("Sub-quadratic options
", "", gr.HTML, {"visible": not native}), "sub_quad_q_chunk_size": OptionInfo(512, "Attention query chunk size", gr.Slider, {"minimum": 16, "maximum": 8192, "step": 8, "visible": not native}), "sub_quad_kv_chunk_size": OptionInfo(512, "Attention kv chunk size", gr.Slider, {"minimum": 0, "maximum": 8192, "step": 8, "visible": not native}), "sub_quad_chunk_threshold": OptionInfo(80, "Attention chunking threshold", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1, "visible": not native}), +})) - "other_sep": OptionInfo("Execution options
", "", gr.HTML), - "opt_channelslast": OptionInfo(False, "Use channels last "), - "cudnn_deterministic": OptionInfo(False, "Use deterministic mode"), - "cudnn_benchmark": OptionInfo(False, "Full-depth cuDNN benchmark feature"), +options_templates.update(options_section(('backends', "Backend Settings"), { + "other_sep": OptionInfo("Torch Options
", "", gr.HTML), + "opt_channelslast": OptionInfo(False, "Channels last "), + "cudnn_deterministic": OptionInfo(False, "Deterministic mode"), + "cudnn_benchmark": OptionInfo(False, "Full-depth cuDNN benchmark"), "diffusers_fuse_projections": OptionInfo(False, "Fused projections"), - "torch_expandable_segments": OptionInfo(False, "Torch expandable segments"), - "cuda_mem_fraction": OptionInfo(0.0, "Torch memory limit", gr.Slider, {"minimum": 0, "maximum": 2.0, "step": 0.05}), - "torch_gc_threshold": OptionInfo(80, "Torch memory threshold for GC", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1}), - "torch_malloc": OptionInfo("native", "Torch memory allocator", gr.Radio, {"choices": ['native', 'cudaMallocAsync'] }), + "torch_expandable_segments": OptionInfo(False, "Expandable segments"), + "cuda_mem_fraction": OptionInfo(0.0, "Memory limit", gr.Slider, {"minimum": 0, "maximum": 2.0, "step": 0.05}), + "torch_gc_threshold": OptionInfo(70, "GC threshold", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1}), + "inference_mode": OptionInfo("no-grad", "Inference mode", gr.Radio, {"choices": ["no-grad", "inference-mode", "none"]}), + "torch_malloc": OptionInfo("native", "Memory allocator", gr.Radio, {"choices": ['native', 'cudaMallocAsync'] }), + + "onnx_sep": OptionInfo("ONNX
", "", gr.HTML), + "onnx_execution_provider": OptionInfo(execution_providers.get_default_execution_provider().value, 'ONNX Execution Provider', gr.Dropdown, lambda: {"choices": execution_providers.available_execution_providers }), + "onnx_cpu_fallback": OptionInfo(True, 'ONNX allow fallback to CPU'), + "onnx_cache_converted": OptionInfo(True, 'ONNX cache converted models'), + "onnx_unload_base": OptionInfo(False, 'ONNX unload base model when processing refiner'), - "cuda_compile_sep": OptionInfo("Model Compile
", "", gr.HTML), - "cuda_compile": OptionInfo([] if not cmd_opts.use_openvino else ["Model", "VAE"], "Compile Model", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "Upscaler"]}), - "cuda_compile_backend": OptionInfo("none" if not cmd_opts.use_openvino else "openvino_fx", "Model compile backend", gr.Radio, {"choices": ['none', 'inductor', 'cudagraphs', 'aot_ts_nvfuser', 'hidet', 'migraphx', 'ipex', 'onediff', 'stable-fast', 'deep-cache', 'olive-ai', 'openvino_fx']}), - "cuda_compile_mode": OptionInfo("default", "Model compile mode", gr.Radio, {"choices": ['default', 'reduce-overhead', 'max-autotune', 'max-autotune-no-cudagraphs']}), - "cuda_compile_fullgraph": OptionInfo(True if not cmd_opts.use_openvino else False, "Model compile fullgraph"), - "cuda_compile_precompile": OptionInfo(False, "Model compile precompile"), - "cuda_compile_verbose": OptionInfo(False, "Model compile verbose mode"), - "cuda_compile_errors": OptionInfo(True, "Model compile suppress errors"), - "deep_cache_interval": OptionInfo(3, "DeepCache cache interval", gr.Slider, {"minimum": 1, "maximum": 10, "step": 1}), + "olive_sep": OptionInfo("Olive
", "", gr.HTML), + "olive_float16": OptionInfo(True, 'Olive use FP16 on optimization'), + "olive_vae_encoder_float32": OptionInfo(False, 'Olive force FP32 for VAE Encoder'), + "olive_static_dims": OptionInfo(True, 'Olive use static dimensions'), + "olive_cache_optimized": OptionInfo(True, 'Olive cache optimized models'), "ipex_sep": OptionInfo("IPEX
", "", gr.HTML, {"visible": devices.backend == "ipex"}), - "ipex_optimize": OptionInfo([], "IPEX Optimize for Intel GPUs", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "Upscaler"], "visible": devices.backend == "ipex"}), + "ipex_optimize": OptionInfo([], "IPEX Optimize", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "Upscaler"], "visible": devices.backend == "ipex"}), "openvino_sep": OptionInfo("OpenVINO
", "", gr.HTML, {"visible": cmd_opts.use_openvino}), "openvino_devices": OptionInfo([], "OpenVINO devices to use", gr.CheckboxGroup, {"choices": get_openvino_device_list() if cmd_opts.use_openvino else [], "visible": cmd_opts.use_openvino}), # pylint: disable=E0606 "openvino_accuracy": OptionInfo("performance", "OpenVINO accuracy mode", gr.Radio, {"choices": ['performance', 'accuracy'], "visible": cmd_opts.use_openvino}), - "openvino_disable_model_caching": OptionInfo(False, "OpenVINO disable model caching", gr.Checkbox, {"visible": cmd_opts.use_openvino}), + "openvino_disable_model_caching": OptionInfo(True, "OpenVINO disable model caching", gr.Checkbox, {"visible": cmd_opts.use_openvino}), "openvino_disable_memory_cleanup": OptionInfo(True, "OpenVINO disable memory cleanup after compile", gr.Checkbox, {"visible": cmd_opts.use_openvino}), "directml_sep": OptionInfo("DirectML
", "", gr.HTML, {"visible": devices.backend == "directml"}), "directml_memory_provider": OptionInfo(default_memory_provider, 'DirectML memory stats provider', gr.Radio, {"choices": memory_providers, "visible": devices.backend == "directml"}), "directml_catch_nan": OptionInfo(False, "DirectML retry ops for NaN", gr.Checkbox, {"visible": devices.backend == "directml"}), - - "olive_sep": OptionInfo("Olive
", "", gr.HTML), - "olive_float16": OptionInfo(True, 'Olive use FP16 on optimization'), - "olive_vae_encoder_float32": OptionInfo(False, 'Olive force FP32 for VAE Encoder'), - "olive_static_dims": OptionInfo(True, 'Olive use static dimensions'), - "olive_cache_optimized": OptionInfo(True, 'Olive cache optimized models'), -})) - -options_templates.update(options_section(('diffusers', "Diffusers Settings"), { - "diffusers_pipeline": OptionInfo('Autodetect', 'Diffusers pipeline', gr.Dropdown, lambda: {"choices": list(shared_items.get_pipelines()) }), - "diffuser_cache_config": OptionInfo(True, "Use cached model config when available"), - "diffusers_move_base": OptionInfo(False, "Move base model to CPU when using refiner"), - "diffusers_move_unet": OptionInfo(False, "Move base model to CPU when using VAE"), - "diffusers_move_refiner": OptionInfo(False, "Move refiner model to CPU when not in use"), - "diffusers_extract_ema": OptionInfo(False, "Use model EMA weights when possible"), - "diffusers_generator_device": OptionInfo("GPU", "Generator device", gr.Radio, {"choices": ["GPU", "CPU", "Unset"]}), - "diffusers_offload_mode": OptionInfo(startup_offload_mode, "Model offload mode", gr.Radio, {"choices": ['none', 'balanced', 'model', 'sequential']}), - "diffusers_offload_max_gpu_memory": OptionInfo(round(gpu_memory * 0.75, 1), "Max GPU memory for balanced offload mode in GB", gr.Slider, {"minimum": 0, "maximum": gpu_memory, "step": 0.01,}), - "diffusers_offload_max_cpu_memory": OptionInfo(round(cpu_memory * 0.75, 1), "Max CPU memory for balanced offload mode in GB", gr.Slider, {"minimum": 0, "maximum": cpu_memory, "step": 0.01,}), - "diffusers_vae_upcast": OptionInfo("default", "VAE upcasting", gr.Radio, {"choices": ['default', 'true', 'false']}), - "diffusers_vae_slicing": OptionInfo(True, "VAE slicing"), - "diffusers_vae_tiling": OptionInfo(cmd_opts.lowvram or cmd_opts.medvram, "VAE tiling"), - "diffusers_model_load_variant": OptionInfo("default", "Preferred Model variant", gr.Radio, {"choices": ['default', 'fp32', 'fp16']}), - "diffusers_vae_load_variant": OptionInfo("default", "Preferred VAE variant", gr.Radio, {"choices": ['default', 'fp32', 'fp16']}), - "custom_diffusers_pipeline": OptionInfo('', 'Load custom Diffusers pipeline'), - "diffusers_eval": OptionInfo(True, "Force model eval"), - "diffusers_to_gpu": OptionInfo(False, "Load model directly to GPU"), - "disable_accelerate": OptionInfo(False, "Disable accelerate"), - "diffusers_pooled": OptionInfo("default", "Diffusers SDXL pooled embeds", gr.Radio, {"choices": ['default', 'weighted']}), - "diffusers_zeros_prompt_pad": OptionInfo(False, "Use zeros for prompt padding", gr.Checkbox), - "huggingface_token": OptionInfo('', 'HuggingFace token'), - "enable_linfusion": OptionInfo(False, "Apply LinFusion distillation on load"), - - "onnx_sep": OptionInfo("ONNX Runtime
", "", gr.HTML), - "onnx_execution_provider": OptionInfo(execution_providers.get_default_execution_provider().value, 'Execution Provider', gr.Dropdown, lambda: {"choices": execution_providers.available_execution_providers }), - "onnx_cpu_fallback": OptionInfo(True, 'ONNX allow fallback to CPU'), - "onnx_cache_converted": OptionInfo(True, 'ONNX cache converted models'), - "onnx_unload_base": OptionInfo(False, 'ONNX unload base model when processing refiner'), })) options_templates.update(options_section(('quantization', "Quantization Settings"), { - "bnb_quantization": OptionInfo([], "BnB quantization enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder"], "visible": native}), - "bnb_quantization_type": OptionInfo("nf4", "BnB quantization type", gr.Radio, {"choices": ['nf4', 'fp8', 'fp4'], "visible": native}), - "bnb_quantization_storage": OptionInfo("uint8", "BnB quantization storage", gr.Radio, {"choices": ["float16", "float32", "int8", "uint8", "float64", "bfloat16"], "visible": native}), - "optimum_quanto_weights": OptionInfo([], "Optimum.quanto quantization enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "ControlNet"], "visible": native}), - "optimum_quanto_weights_type": OptionInfo("qint8", "Optimum.quanto quantization type", gr.Radio, {"choices": ['qint8', 'qfloat8_e4m3fn', 'qfloat8_e5m2', 'qint4', 'qint2'], "visible": native}), - "optimum_quanto_activations_type": OptionInfo("none", "Optimum.quanto quantization activations ", gr.Radio, {"choices": ['none', 'qint8', 'qfloat8_e4m3fn', 'qfloat8_e5m2'], "visible": native}), - "torchao_quantization": OptionInfo([], "TorchAO quantization enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder"], "visible": native}), - "torchao_quantization_type": OptionInfo("int8", "TorchAO quantization type", gr.Radio, {"choices": ["int8+act", "int8", "int4", "fp8+act", "fp8", "fpx"], "visible": native}), - "nncf_compress_weights": OptionInfo([], "NNCF compression enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "ControlNet"], "visible": native}), - "nncf_compress_weights_mode": OptionInfo("INT8", "NNCF compress mode", gr.Radio, {"choices": ['INT8', 'INT8_SYM', 'INT4_ASYM', 'INT4_SYM', 'NF4'] if cmd_opts.use_openvino else ['INT8']}), - "nncf_compress_weights_raito": OptionInfo(1.0, "NNCF compress ratio", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01, "visible": cmd_opts.use_openvino}), - "nncf_quantize": OptionInfo([], "NNCF OpenVINO quantization enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder"], "visible": cmd_opts.use_openvino}), - "nncf_quant_mode": OptionInfo("INT8", "NNCF OpenVINO quantization mode", gr.Radio, {"choices": ['INT8', 'FP8_E4M3', 'FP8_E5M2'], "visible": cmd_opts.use_openvino}), - - "quant_shuffle_weights": OptionInfo(False, "Shuffle the weights between GPU and CPU when quantizing", gr.Checkbox, {"visible": native}), + "bnb_sep": OptionInfo("BitsAndBytes
", "", gr.HTML), + "bnb_quantization": OptionInfo([], "Quantization enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder"], "visible": native}), + "bnb_quantization_type": OptionInfo("nf4", "Quantization type", gr.Dropdown, {"choices": ['nf4', 'fp8', 'fp4'], "visible": native}), + "bnb_quantization_storage": OptionInfo("uint8", "Backend storage", gr.Dropdown, {"choices": ["float16", "float32", "int8", "uint8", "float64", "bfloat16"], "visible": native}), + "optimum_quanto_sep": OptionInfo("Optimum Quanto
", "", gr.HTML), + "optimum_quanto_weights": OptionInfo([], "Quantization enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "ControlNet"], "visible": native}), + "optimum_quanto_weights_type": OptionInfo("qint8", "Quantization weights type", gr.Dropdown, {"choices": ['qint8', 'qfloat8_e4m3fn', 'qfloat8_e5m2', 'qint4', 'qint2'], "visible": native}), + "optimum_quanto_activations_type": OptionInfo("none", "Quantization activations type ", gr.Dropdown, {"choices": ['none', 'qint8', 'qfloat8_e4m3fn', 'qfloat8_e5m2'], "visible": native}), + "torchao_sep": OptionInfo("TorchAO
", "", gr.HTML), + "torchao_quantization": OptionInfo([], "Quantization enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder"], "visible": native}), + "torchao_quantization_mode": OptionInfo("pre", "Quantization mode", gr.Dropdown, {"choices": ['pre', 'post'], "visible": native}), + "torchao_quantization_type": OptionInfo("int8_weight_only", "Quantization type", gr.Dropdown, {"choices": ['int4_weight_only', 'int8_dynamic_activation_int4_weight', 'int8_weight_only', 'int8_dynamic_activation_int8_weight', 'float8_weight_only', 'float8_dynamic_activation_float8_weight', 'float8_static_activation_float8_weight'], "visible": native}), + "nncf_sep": OptionInfo("NNCF
", "", gr.HTML), + "nncf_compress_weights": OptionInfo([], "Quantization enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "ControlNet"], "visible": native}), + "nncf_compress_weights_mode": OptionInfo("INT8", "Quantization type", gr.Dropdown, {"choices": ['INT8', 'INT8_SYM', 'INT4_ASYM', 'INT4_SYM', 'NF4'] if cmd_opts.use_openvino else ['INT8']}), + "nncf_compress_weights_raito": OptionInfo(1.0, "Compress ratio", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01, "visible": cmd_opts.use_openvino}), + "nncf_quantize": OptionInfo([], "OpenVINO enabled", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder"], "visible": cmd_opts.use_openvino}), + "nncf_quant_mode": OptionInfo("INT8", "OpenVINO mode", gr.Dropdown, {"choices": ['INT8', 'FP8_E4M3', 'FP8_E5M2'], "visible": cmd_opts.use_openvino}), + "quant_shuffle_weights": OptionInfo(False, "Shuffle weights", gr.Checkbox, {"visible": native}), })) -options_templates.update(options_section(('advanced', "Inference Settings"), { - "token_merging_sep": OptionInfo("Token merging
", "", gr.HTML), +options_templates.update(options_section(('advanced', "Pipeline Modifiers"), { + "token_merging_sep": OptionInfo("Token Merging
", "", gr.HTML), "token_merging_method": OptionInfo("None", "Token merging method", gr.Radio, {"choices": ['None', 'ToMe', 'ToDo']}), "tome_ratio": OptionInfo(0.0, "ToMe token merging ratio", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.05}), "todo_ratio": OptionInfo(0.0, "ToDo token merging ratio", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.05}), "freeu_sep": OptionInfo("FreeU
", "", gr.HTML), "freeu_enabled": OptionInfo(False, "FreeU"), - "freeu_b1": OptionInfo(1.2, "1st stage backbone factor", gr.Slider, {"minimum": 1.0, "maximum": 2.0, "step": 0.01}), - "freeu_b2": OptionInfo(1.4, "2nd stage backbone factor", gr.Slider, {"minimum": 1.0, "maximum": 2.0, "step": 0.01}), - "freeu_s1": OptionInfo(0.9, "1st stage skip factor", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01}), - "freeu_s2": OptionInfo(0.2, "2nd stage skip factor", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01}), + "freeu_b1": OptionInfo(1.2, "1st stage backbone", gr.Slider, {"minimum": 1.0, "maximum": 2.0, "step": 0.01}), + "freeu_b2": OptionInfo(1.4, "2nd stage backbone", gr.Slider, {"minimum": 1.0, "maximum": 2.0, "step": 0.01}), + "freeu_s1": OptionInfo(0.9, "1st stage skip", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01}), + "freeu_s2": OptionInfo(0.2, "2nd stage skip", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01}), "pag_sep": OptionInfo("Perturbed-Attention Guidance
", "", gr.HTML), "pag_apply_layers": OptionInfo("m0", "PAG layer names"), "hypertile_sep": OptionInfo("HyperTile
", "", gr.HTML), - "hypertile_hires_only": OptionInfo(False, "HyperTile hires pass only"), - "hypertile_unet_enabled": OptionInfo(False, "HyperTile UNet"), - "hypertile_unet_tile": OptionInfo(0, "HyperTile UNet tile size", gr.Slider, {"minimum": 0, "maximum": 1024, "step": 8}), - "hypertile_unet_swap_size": OptionInfo(1, "HyperTile UNet swap size", gr.Slider, {"minimum": 1, "maximum": 10, "step": 1}), - "hypertile_unet_depth": OptionInfo(0, "HyperTile UNet depth", gr.Slider, {"minimum": 0, "maximum": 4, "step": 1}), - "hypertile_vae_enabled": OptionInfo(False, "HyperTile VAE", gr.Checkbox), - "hypertile_vae_tile": OptionInfo(128, "HyperTile VAE tile size", gr.Slider, {"minimum": 0, "maximum": 1024, "step": 8}), - "hypertile_vae_swap_size": OptionInfo(1, "HyperTile VAE swap size", gr.Slider, {"minimum": 1, "maximum": 10, "step": 1}), + "hypertile_hires_only": OptionInfo(False, "HiRes pass only"), + "hypertile_unet_enabled": OptionInfo(False, "UNet Enabled"), + "hypertile_unet_tile": OptionInfo(0, "UNet tile size", gr.Slider, {"minimum": 0, "maximum": 1024, "step": 8}), + "hypertile_unet_swap_size": OptionInfo(1, "UNet swap size", gr.Slider, {"minimum": 1, "maximum": 10, "step": 1}), + "hypertile_unet_depth": OptionInfo(0, "UNet depth", gr.Slider, {"minimum": 0, "maximum": 4, "step": 1}), + "hypertile_vae_enabled": OptionInfo(False, "VAE Enabled", gr.Checkbox), + "hypertile_vae_tile": OptionInfo(128, "VAE tile size", gr.Slider, {"minimum": 0, "maximum": 1024, "step": 8}), + "hypertile_vae_swap_size": OptionInfo(1, "VAE swap size", gr.Slider, {"minimum": 1, "maximum": 10, "step": 1}), "hidiffusion_sep": OptionInfo("HiDiffusion
", "", gr.HTML), "hidiffusion_raunet": OptionInfo(True, "Apply RAU-Net"), @@ -633,16 +636,28 @@ def get_default_modes(): "hidiffusion_t1": OptionInfo(-1, "Override T1 ratio", gr.Slider, {"minimum": -1, "maximum": 1.0, "step": 0.05}), "hidiffusion_t2": OptionInfo(-1, "Override T2 ratio", gr.Slider, {"minimum": -1, "maximum": 1.0, "step": 0.05}), + "linfusion_sep": OptionInfo("Batch
", "", gr.HTML), + "enable_linfusion": OptionInfo(False, "Apply LinFusion distillation on load"), + "inference_batch_sep": OptionInfo("Batch
", "", gr.HTML), "sequential_seed": OptionInfo(True, "Batch mode uses sequential seeds"), "batch_frame_mode": OptionInfo(False, "Parallel process images in batch"), - "inference_other_sep": OptionInfo("Other
", "", gr.HTML), - "inference_mode": OptionInfo("no-grad", "Torch inference mode", gr.Radio, {"choices": ["no-grad", "inference-mode", "none"]}), - "sd_vae_sliced_encode": OptionInfo(False, "VAE sliced encode", gr.Checkbox, {"visible": not native}), +})) + +options_templates.update(options_section(('compile', "Model Compile"), { + "cuda_compile_sep": OptionInfo("Model Compile
", "", gr.HTML), + "cuda_compile": OptionInfo([] if not cmd_opts.use_openvino else ["Model", "VAE"], "Compile Model", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "Upscaler"]}), + "cuda_compile_backend": OptionInfo("none" if not cmd_opts.use_openvino else "openvino_fx", "Model compile backend", gr.Radio, {"choices": ['none', 'inductor', 'cudagraphs', 'aot_ts_nvfuser', 'hidet', 'migraphx', 'ipex', 'onediff', 'stable-fast', 'deep-cache', 'olive-ai', 'openvino_fx']}), + "cuda_compile_mode": OptionInfo("default", "Model compile mode", gr.Radio, {"choices": ['default', 'reduce-overhead', 'max-autotune', 'max-autotune-no-cudagraphs']}), + "cuda_compile_fullgraph": OptionInfo(True if not cmd_opts.use_openvino else False, "Model compile fullgraph"), + "cuda_compile_precompile": OptionInfo(False, "Model compile precompile"), + "cuda_compile_verbose": OptionInfo(False, "Model compile verbose mode"), + "cuda_compile_errors": OptionInfo(True, "Model compile suppress errors"), + "deep_cache_interval": OptionInfo(3, "DeepCache cache interval", gr.Slider, {"minimum": 1, "maximum": 10, "step": 1}), })) options_templates.update(options_section(('system-paths', "System Paths"), { - "models_paths_sep_options": OptionInfo("Models paths
", "", gr.HTML), + "models_paths_sep_options": OptionInfo("Models Paths
", "", gr.HTML), "models_dir": OptionInfo('models', "Base path where all models are stored", folder=True), "ckpt_dir": OptionInfo(os.path.join(paths.models_path, 'Stable-diffusion'), "Folder with stable diffusion models", folder=True), "diffusers_dir": OptionInfo(os.path.join(paths.models_path, 'Diffusers'), "Folder with Huggingface models", folder=True), @@ -723,13 +738,13 @@ def get_default_modes(): })) options_templates.update(options_section(('saving-paths', "Image Paths"), { - "saving_sep_images": OptionInfo("Save options
", "", gr.HTML), + "saving_sep_images": OptionInfo("Save Options
", "", gr.HTML), "save_images_add_number": OptionInfo(True, "Numbered filenames", component_args=hide_dirs), "use_original_name_batch": OptionInfo(True, "Batch uses original name"), "save_to_dirs": OptionInfo(False, "Save images to a subdirectory"), "directories_filename_pattern": OptionInfo("[date]", "Directory name pattern", component_args=hide_dirs), "samples_filename_pattern": OptionInfo("[seq]-[model_name]-[prompt_words]", "Images filename pattern", component_args=hide_dirs), - "directories_max_prompt_words": OptionInfo(8, "Max words per pattern", gr.Slider, {"minimum": 1, "maximum": 99, "step": 1, **hide_dirs}), + "directories_max_prompt_words": OptionInfo(8, "Max words", gr.Slider, {"minimum": 1, "maximum": 99, "step": 1, **hide_dirs}), "outdir_sep_dirs": OptionInfo("Folders
", "", gr.HTML), "outdir_samples": OptionInfo("", "Images folder", component_args=hide_dirs, folder=True), @@ -748,14 +763,14 @@ def get_default_modes(): "outdir_control_grids": OptionInfo("outputs/grids", 'Folder for control grids', component_args=hide_dirs, folder=True), })) -options_templates.update(options_section(('ui', "User Interface Options"), { +options_templates.update(options_section(('ui', "User Interface"), { "theme_type": OptionInfo("Standard", "Theme type", gr.Radio, {"choices": ["Modern", "Standard", "None"]}), "theme_style": OptionInfo("Auto", "Theme mode", gr.Radio, {"choices": ["Auto", "Dark", "Light"]}), "gradio_theme": OptionInfo("black-teal", "UI theme", gr.Dropdown, lambda: {"choices": theme.list_themes()}, refresh=theme.refresh_themes), "autolaunch": OptionInfo(False, "Autolaunch browser upon startup"), "font_size": OptionInfo(14, "Font size", gr.Slider, {"minimum": 8, "maximum": 32, "step": 1, "visible": True}), "aspect_ratios": OptionInfo("1:1, 4:3, 3:2, 16:9, 16:10, 21:9, 2:3, 3:4, 9:16, 10:16, 9:21", "Allowed aspect ratios"), - "motd": OptionInfo(True, "Show MOTD"), + "motd": OptionInfo(False, "Show MOTD"), "compact_view": OptionInfo(False, "Compact view"), "return_grid": OptionInfo(True, "Show grid in results"), "return_mask": OptionInfo(False, "Inpainting include greyscale mask in results"), @@ -767,14 +782,15 @@ def get_default_modes(): })) options_templates.update(options_section(('live-preview', "Live Previews"), { - "notification_audio_enable": OptionInfo(False, "Play a notification upon completion"), - "notification_audio_path": OptionInfo("html/notification.mp3","Path to notification sound", component_args=hide_dirs, folder=True), "show_progress_every_n_steps": OptionInfo(1, "Live preview display period", gr.Slider, {"minimum": 0, "maximum": 32, "step": 1}), "show_progress_type": OptionInfo("Approximate", "Live preview method", gr.Radio, {"choices": ["Simple", "Approximate", "TAESD", "Full VAE"]}), "live_preview_refresh_period": OptionInfo(500, "Progress update period", gr.Slider, {"minimum": 0, "maximum": 5000, "step": 25}), "live_preview_taesd_layers": OptionInfo(3, "TAESD decode layers", gr.Slider, {"minimum": 1, "maximum": 3, "step": 1}), + "live_preview_downscale": OptionInfo(True, "Downscale high resolution live previews"), "logmonitor_show": OptionInfo(True, "Show log view"), "logmonitor_refresh_period": OptionInfo(5000, "Log view update period", gr.Slider, {"minimum": 0, "maximum": 30000, "step": 25}), + "notification_audio_enable": OptionInfo(False, "Play a notification upon completion"), + "notification_audio_path": OptionInfo("html/notification.mp3","Path to notification sound", component_args=hide_dirs, folder=True), })) options_templates.update(options_section(('sampler-params', "Sampler Settings"), { @@ -813,7 +829,7 @@ def get_default_modes(): 's_noise': OptionInfo(1.0, "Sigma noise", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01, "visible": not native}), 's_min': OptionInfo(0.0, "Sigma min", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01, "visible": not native}), 's_max': OptionInfo(0.0, "Sigma max", gr.Slider, {"minimum": 0.0, "maximum": 100.0, "step": 1.0, "visible": not native}), - "schedulers_sep_compvis": OptionInfo("CompVis specific config
", "", gr.HTML, {"visible": not native}), + "schedulers_sep_compvis": OptionInfo("CompVis Config
", "", gr.HTML, {"visible": not native}), 'uni_pc_variant': OptionInfo("bh2", "UniPC variant", gr.Radio, {"choices": ["bh1", "bh2", "vary_coeff"], "visible": not native}), 'uni_pc_skip_type': OptionInfo("time_uniform", "UniPC skip type", gr.Radio, {"choices": ["time_uniform", "time_quadratic", "logSNR"], "visible": not native}), "ddim_discretize": OptionInfo('uniform', "DDIM discretize img2img", gr.Radio, {"choices": ['uniform', 'quad'], "visible": not native}), @@ -821,9 +837,9 @@ def get_default_modes(): options_templates.update(options_section(('postprocessing', "Postprocessing"), { 'postprocessing_enable_in_main_ui': OptionInfo([], "Additional postprocessing operations", gr.Dropdown, lambda: {"multiselect":True, "choices": [x.name for x in shared_items.postprocessing_scripts()]}), - 'postprocessing_operation_order': OptionInfo([], "Postprocessing operation order", gr.Dropdown, lambda: {"multiselect":True, "choices": [x.name for x in shared_items.postprocessing_scripts()]}), + 'postprocessing_operation_order': OptionInfo([], "Postprocessing operation order", gr.Dropdown, lambda: {"multiselect":True, "choices": [x.name for x in shared_items.postprocessing_scripts()], "visible": False }), - "postprocessing_sep_img2img": OptionInfo("Img2Img & Inpainting
", "", gr.HTML), + "postprocessing_sep_img2img": OptionInfo("Inpaint
", "", gr.HTML), "img2img_color_correction": OptionInfo(False, "Apply color correction"), "mask_apply_overlay": OptionInfo(True, "Apply mask as overlay"), "img2img_background_color": OptionInfo("#ffffff", "Image transparent color fill", gr.ColorPicker, {}), @@ -831,7 +847,7 @@ def get_default_modes(): "initial_noise_multiplier": OptionInfo(1.0, "Noise multiplier for image processing", gr.Slider, {"minimum": 0.1, "maximum": 1.5, "step": 0.01, "visible": not native}), "img2img_extra_noise": OptionInfo(0.0, "Extra noise multiplier for img2img", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01, "visible": not native}), - # "postprocessing_sep_detailer": OptionInfo("Detailer
", "", gr.HTML), + "postprocessing_sep_detailer": OptionInfo("Detailer
", "", gr.HTML), "detailer_model": OptionInfo("Detailer", "Detailer model", gr.Radio, lambda: {"choices": [x.name() for x in detailers], "visible": False}), "detailer_classes": OptionInfo("", "Detailer classes", gr.Textbox, { "visible": False}), "detailer_conf": OptionInfo(0.6, "Min confidence", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.05, "visible": False}), @@ -843,11 +859,12 @@ def get_default_modes(): "detailer_blur": OptionInfo(10, "Item edge blur", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1, "visible": False}), "detailer_strength": OptionInfo(0.5, "Detailer strength", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01, "visible": False}), "detailer_models": OptionInfo(['face-yolo8n'], "Detailer models", gr.Dropdown, lambda: {"multiselect":True, "choices": list(yolo.list), "visible": False}), - "code_former_weight": OptionInfo(0.2, "CodeFormer weight parameter", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01, "visible": False}), "detailer_unload": OptionInfo(False, "Move detailer model to CPU when complete"), + "detailer_augment": OptionInfo(True, "Detailer use model augment"), - "postprocessing_sep_face_restore": OptionInfo("Face restore
", "", gr.HTML), - "face_restoration_model": OptionInfo("Face restorer", "Face restoration", gr.Radio, lambda: {"choices": ['None'] + [x.name() for x in face_restorers]}), + "postprocessing_sep_face_restore": OptionInfo("Face Restore
", "", gr.HTML), + "face_restoration_model": OptionInfo("None", "Face restoration", gr.Radio, lambda: {"choices": ['None'] + [x.name() for x in face_restorers]}), + "code_former_weight": OptionInfo(0.2, "CodeFormer weight parameter", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01}), "postprocessing_sep_upscalers": OptionInfo("Upscaling
", "", gr.HTML), "upscaler_unload": OptionInfo(False, "Unload upscaler after processing"), @@ -857,6 +874,7 @@ def get_default_modes(): options_templates.update(options_section(('control', "Control Options"), { "control_max_units": OptionInfo(4, "Maximum number of units", gr.Slider, {"minimum": 1, "maximum": 10, "step": 1}), + "control_tiles": OptionInfo("1x1, 1x2, 1x3, 1x4, 2x1, 2x1, 2x2, 2x3, 2x4, 3x1, 3x2, 3x3, 3x4, 4x1, 4x2, 4x3, 4x4", "Tiling options"), "control_move_processor": OptionInfo(False, "Processor move to CPU after use"), "control_unload_processor": OptionInfo(False, "Processor unload after use"), })) @@ -875,6 +893,15 @@ def get_default_modes(): "deepbooru_filter_tags": OptionInfo("", "Filter out tags from deepbooru output"), })) +options_templates.update(options_section(('huggingface', "Huggingface"), { + "huggingface_sep": OptionInfo("Huggingface
", "", gr.HTML), + "diffuser_cache_config": OptionInfo(True, "Use cached model config when available"), + "huggingface_token": OptionInfo('', 'HuggingFace token'), + "diffusers_model_load_variant": OptionInfo("default", "Preferred Model variant", gr.Radio, {"choices": ['default', 'fp32', 'fp16']}), + "diffusers_vae_load_variant": OptionInfo("default", "Preferred VAE variant", gr.Radio, {"choices": ['default', 'fp32', 'fp16']}), + "custom_diffusers_pipeline": OptionInfo('', 'Load custom Diffusers pipeline'), +})) + options_templates.update(options_section(('extra_networks', "Networks"), { "extra_networks_sep1": OptionInfo("Networks UI
", "", gr.HTML), "extra_networks_show": OptionInfo(True, "UI show on startup"), @@ -891,23 +918,27 @@ def get_default_modes(): "extra_networks_model_sep": OptionInfo("Models
", "", gr.HTML), "extra_network_reference": OptionInfo(False, "Use reference values when available", gr.Checkbox), - "extra_networks_embed_sep": OptionInfo("Embeddings
", "", gr.HTML), - "diffusers_convert_embed": OptionInfo(False, "Auto-convert SD 1.5 embeddings to SDXL ", gr.Checkbox, {"visible": native}), - "extra_networks_styles_sep": OptionInfo("Styles
", "", gr.HTML), - "extra_networks_styles": OptionInfo(True, "Show built-in styles"), - "extra_networks_wildcard_sep": OptionInfo("Wildcards
", "", gr.HTML), - "wildcards_enabled": OptionInfo(True, "Enable file wildcards support"), + "extra_networks_lora_sep": OptionInfo("LoRA
", "", gr.HTML), "extra_networks_default_multiplier": OptionInfo(1.0, "Default strength", gr.Slider, {"minimum": 0.0, "maximum": 2.0, "step": 0.01}), - "lora_preferred_name": OptionInfo("filename", "LoRA preferred name", gr.Radio, {"choices": ["filename", "alias"]}), - "lora_add_hashes_to_infotext": OptionInfo(False, "LoRA add hash info"), + "lora_preferred_name": OptionInfo("filename", "LoRA preferred name", gr.Radio, {"choices": ["filename", "alias"], "visible": False}), + "lora_add_hashes_to_infotext": OptionInfo(False, "LoRA add hash info to metadata"), + "lora_fuse_diffusers": OptionInfo(True, "LoRA fuse directly to model"), "lora_force_diffusers": OptionInfo(False if not cmd_opts.use_openvino else True, "LoRA force loading of all models using Diffusers"), "lora_maybe_diffusers": OptionInfo(False, "LoRA force loading of specific models using Diffusers"), - "lora_fuse_diffusers": OptionInfo(False if not cmd_opts.use_openvino else True, "LoRA use fuse when possible"), "lora_apply_tags": OptionInfo(0, "LoRA auto-apply tags", gr.Slider, {"minimum": -1, "maximum": 32, "step": 1}), "lora_in_memory_limit": OptionInfo(0, "LoRA memory cache", gr.Slider, {"minimum": 0, "maximum": 24, "step": 1}), - "lora_quant": OptionInfo("NF4","LoRA precision in quantized models", gr.Radio, {"choices": ["NF4", "FP4"]}), - "lora_load_gpu": OptionInfo(True if not cmd_opts.lowvram else False, "Load LoRA directly to GPU"), + "lora_quant": OptionInfo("NF4","LoRA precision when quantized", gr.Radio, {"choices": ["NF4", "FP4"]}), + + "extra_networks_styles_sep": OptionInfo("Styles
", "", gr.HTML), + "extra_networks_styles": OptionInfo(True, "Show built-in styles"), + + "extra_networks_embed_sep": OptionInfo("Embeddings
", "", gr.HTML), + "diffusers_enable_embed": OptionInfo(True, "Enable embeddings support", gr.Checkbox, {"visible": native}), + "diffusers_convert_embed": OptionInfo(False, "Auto-convert SD15 embeddings to SDXL", gr.Checkbox, {"visible": native}), + + "extra_networks_wildcard_sep": OptionInfo("Wildcards
", "", gr.HTML), + "wildcards_enabled": OptionInfo(True, "Enable file wildcards support"), })) options_templates.update(options_section((None, "Internal options"), { diff --git a/modules/shared_items.py b/modules/shared_items.py index 4d9b325c8..9abb64718 100644 --- a/modules/shared_items.py +++ b/modules/shared_items.py @@ -86,6 +86,14 @@ def get_pipelines(): 'Kolors': getattr(diffusers, 'KolorsPipeline', None), 'AuraFlow': getattr(diffusers, 'AuraFlowPipeline', None), 'CogView': getattr(diffusers, 'CogView3PlusPipeline', None), + 'Stable Cascade': getattr(diffusers, 'StableCascadeCombinedPipeline', None), + 'PixArt-Sigma': getattr(diffusers, 'PixArtSigmaPipeline', None), + 'HunyuanDiT': getattr(diffusers, 'HunyuanDiTPipeline', None), + 'Stable Diffusion 3': getattr(diffusers, 'StableDiffusion3Pipeline', None), + 'Stable Diffusion 3 Img2Img': getattr(diffusers, 'StableDiffusion3Img2ImgPipeline', None), + 'Lumina-Next': getattr(diffusers, 'LuminaText2ImgPipeline', None), + 'FLUX': getattr(diffusers, 'FluxPipeline', None), + 'Sana': getattr(diffusers, 'SanaPAGPipeline', None), } if hasattr(diffusers, 'OnnxStableDiffusionPipeline'): onnx_pipelines = { @@ -103,19 +111,10 @@ def get_pipelines(): pipelines.update(onnx_pipelines) # items that may rely on diffusers dev version - if hasattr(diffusers, 'StableCascadeCombinedPipeline'): - pipelines['Stable Cascade'] = getattr(diffusers, 'StableCascadeCombinedPipeline', None) - if hasattr(diffusers, 'PixArtSigmaPipeline'): - pipelines['PixArt-Sigma'] = getattr(diffusers, 'PixArtSigmaPipeline', None) - if hasattr(diffusers, 'HunyuanDiTPipeline'): - pipelines['HunyuanDiT'] = getattr(diffusers, 'HunyuanDiTPipeline', None) - if hasattr(diffusers, 'StableDiffusion3Pipeline'): - pipelines['Stable Diffusion 3'] = getattr(diffusers, 'StableDiffusion3Pipeline', None) - pipelines['Stable Diffusion 3 Img2Img'] = getattr(diffusers, 'StableDiffusion3Img2ImgPipeline', None) - if hasattr(diffusers, 'LuminaText2ImgPipeline'): - pipelines['Lumina-Next'] = getattr(diffusers, 'LuminaText2ImgPipeline', None) + """ if hasattr(diffusers, 'FluxPipeline'): pipelines['FLUX'] = getattr(diffusers, 'FluxPipeline', None) + """ for k, v in pipelines.items(): if k != 'Autodetect' and v is None: diff --git a/modules/shared_state.py b/modules/shared_state.py index 7def42b8c..a3312ec33 100644 --- a/modules/shared_state.py +++ b/modules/shared_state.py @@ -28,6 +28,9 @@ class State: oom = False debug_output = os.environ.get('SD_STATE_DEBUG', None) + def __str__(self) -> str: + return f'State: job={self.job} {self.job_no}/{self.job_count} step={self.sampling_step}/{self.sampling_steps} skipped={self.skipped} interrupted={self.interrupted} paused={self.paused} info={self.textinfo}' + def skip(self): log.debug('Requested skip') self.skipped = True @@ -135,11 +138,12 @@ def end(self, api=None): modules.devices.torch_gc() def set_current_image(self): + if self.job == 'VAE': # avoid generating preview while vae is running + return from modules.shared import opts, cmd_opts - """sets self.current_image from self.current_latent if enough sampling steps have been made after the last call to this""" - if cmd_opts.lowvram or self.api: + if cmd_opts.lowvram or self.api or not opts.live_previews_enable or opts.show_progress_every_n_steps <= 0: return - if abs(self.sampling_step - self.current_image_sampling_step) >= opts.show_progress_every_n_steps and opts.live_previews_enable and opts.show_progress_every_n_steps > 0: + if abs(self.sampling_step - self.current_image_sampling_step) >= opts.show_progress_every_n_steps: self.do_set_current_image() def do_set_current_image(self): diff --git a/modules/style_aligned/inversion.py b/modules/style_aligned/inversion.py new file mode 100644 index 000000000..8c91cc02a --- /dev/null +++ b/modules/style_aligned/inversion.py @@ -0,0 +1,124 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from __future__ import annotations +from typing import Callable, TYPE_CHECKING +from diffusers import StableDiffusionXLPipeline +import torch +from tqdm import tqdm +if TYPE_CHECKING: + import numpy as np + + +T = torch.Tensor +TN = T +InversionCallback = Callable[[StableDiffusionXLPipeline, int, T, dict[str, T]], dict[str, T]] + + +def _get_text_embeddings(prompt: str, tokenizer, text_encoder, device): + # Tokenize text and get embeddings + text_inputs = tokenizer(prompt, padding='max_length', max_length=tokenizer.model_max_length, truncation=True, return_tensors='pt') + text_input_ids = text_inputs.input_ids + + with torch.no_grad(): + prompt_embeds = text_encoder( + text_input_ids.to(device), + output_hidden_states=True, + ) + + pooled_prompt_embeds = prompt_embeds[0] + prompt_embeds = prompt_embeds.hidden_states[-2] + if prompt == '': + negative_prompt_embeds = torch.zeros_like(prompt_embeds) + negative_pooled_prompt_embeds = torch.zeros_like(pooled_prompt_embeds) + return negative_prompt_embeds, negative_pooled_prompt_embeds + return prompt_embeds, pooled_prompt_embeds + + +def _encode_text_sdxl(model: StableDiffusionXLPipeline, prompt: str) -> tuple[dict[str, T], T]: + device = model._execution_device # pylint: disable=protected-access + prompt_embeds, pooled_prompt_embeds, = _get_text_embeddings(prompt, model.tokenizer, model.text_encoder, device) # pylint: disable=unused-variable + prompt_embeds_2, pooled_prompt_embeds2, = _get_text_embeddings( prompt, model.tokenizer_2, model.text_encoder_2, device) + prompt_embeds = torch.cat((prompt_embeds, prompt_embeds_2), dim=-1) + text_encoder_projection_dim = model.text_encoder_2.config.projection_dim + add_time_ids = model._get_add_time_ids((1024, 1024), (0, 0), (1024, 1024), model.text_encoder.dtype, # pylint: disable=protected-access + text_encoder_projection_dim).to(device) + added_cond_kwargs = {"text_embeds": pooled_prompt_embeds2, "time_ids": add_time_ids} + return added_cond_kwargs, prompt_embeds + + +def _encode_text_sdxl_with_negative(model: StableDiffusionXLPipeline, prompt: str) -> tuple[dict[str, T], T]: + added_cond_kwargs, prompt_embeds = _encode_text_sdxl(model, prompt) + added_cond_kwargs_uncond, prompt_embeds_uncond = _encode_text_sdxl(model, "") + prompt_embeds = torch.cat((prompt_embeds_uncond, prompt_embeds, )) + added_cond_kwargs = {"text_embeds": torch.cat((added_cond_kwargs_uncond["text_embeds"], added_cond_kwargs["text_embeds"])), + "time_ids": torch.cat((added_cond_kwargs_uncond["time_ids"], added_cond_kwargs["time_ids"])),} + return added_cond_kwargs, prompt_embeds + + +def _encode_image(model: StableDiffusionXLPipeline, image: np.ndarray) -> T: + image = torch.from_numpy(image).float() / 255. + image = (image * 2 - 1).permute(2, 0, 1).unsqueeze(0) + latent = model.vae.encode(image.to(model.vae.device, model.vae.dtype))['latent_dist'].mean * model.vae.config.scaling_factor + return latent + + +def _next_step(model: StableDiffusionXLPipeline, model_output: T, timestep: int, sample: T) -> T: + timestep, next_timestep = min(timestep - model.scheduler.config.num_train_timesteps // model.scheduler.num_inference_steps, 999), timestep + alpha_prod_t = model.scheduler.alphas_cumprod[int(timestep)] if timestep >= 0 else model.scheduler.final_alpha_cumprod + alpha_prod_t_next = model.scheduler.alphas_cumprod[int(next_timestep)] + beta_prod_t = 1 - alpha_prod_t + next_original_sample = (sample - beta_prod_t ** 0.5 * model_output) / alpha_prod_t ** 0.5 + next_sample_direction = (1 - alpha_prod_t_next) ** 0.5 * model_output + next_sample = alpha_prod_t_next ** 0.5 * next_original_sample + next_sample_direction + return next_sample + + +def _get_noise_pred(model: StableDiffusionXLPipeline, latent: T, t: T, context: T, guidance_scale: float, added_cond_kwargs: dict[str, T]): + latents_input = torch.cat([latent] * 2) + noise_pred = model.unet(latents_input, t, encoder_hidden_states=context, added_cond_kwargs=added_cond_kwargs)["sample"] + noise_pred_uncond, noise_prediction_text = noise_pred.chunk(2) + noise_pred = noise_pred_uncond + guidance_scale * (noise_prediction_text - noise_pred_uncond) + # latents = next_step(model, noise_pred, t, latent) + return noise_pred + + +def _ddim_loop(model: StableDiffusionXLPipeline, z0, prompt, guidance_scale) -> T: + all_latent = [z0] + added_cond_kwargs, text_embedding = _encode_text_sdxl_with_negative(model, prompt) + latent = z0.clone().detach().to(model.text_encoder.dtype) + for i in tqdm(range(model.scheduler.num_inference_steps)): + t = model.scheduler.timesteps[len(model.scheduler.timesteps) - i - 1] + noise_pred = _get_noise_pred(model, latent, t, text_embedding, guidance_scale, added_cond_kwargs) + latent = _next_step(model, noise_pred, t, latent) + all_latent.append(latent) + return torch.cat(all_latent).flip(0) + + +def make_inversion_callback(zts, offset: int = 0): + + def callback_on_step_end(pipeline: StableDiffusionXLPipeline, i: int, t: T, callback_kwargs: dict[str, T]) -> dict[str, T]: # pylint: disable=unused-argument + latents = callback_kwargs['latents'] + latents[0] = zts[max(offset + 1, i + 1)].to(latents.device, latents.dtype) + return {'latents': latents} + return zts[offset], callback_on_step_end + + +@torch.no_grad() +def ddim_inversion(model: StableDiffusionXLPipeline, x0: np.ndarray, prompt: str, num_inference_steps: int, guidance_scale,) -> T: + z0 = _encode_image(model, x0) + model.scheduler.set_timesteps(num_inference_steps, device=z0.device) + zs = _ddim_loop(model, z0, prompt, guidance_scale) + return zs diff --git a/modules/style_aligned/sa_handler.py b/modules/style_aligned/sa_handler.py new file mode 100644 index 000000000..ee4b1ca79 --- /dev/null +++ b/modules/style_aligned/sa_handler.py @@ -0,0 +1,281 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from __future__ import annotations +from typing import TYPE_CHECKING +if TYPE_CHECKING: + from diffusers import StableDiffusionXLPipeline +from dataclasses import dataclass +import torch +import torch.nn as nn +from torch.nn import functional as nnf +from diffusers.models import attention_processor # pylint: disable=ungrouped-imports +import einops + +T = torch.Tensor + + +@dataclass(frozen=True) +class StyleAlignedArgs: + share_group_norm: bool = True + share_layer_norm: bool = True + share_attention: bool = True + adain_queries: bool = True + adain_keys: bool = True + adain_values: bool = False + full_attention_share: bool = False + shared_score_scale: float = 1. + shared_score_shift: float = 0. + only_self_level: float = 0. + + +def expand_first(feat: T, scale=1.,) -> T: + b = feat.shape[0] + feat_style = torch.stack((feat[0], feat[b // 2])).unsqueeze(1) + if scale == 1: + feat_style = feat_style.expand(2, b // 2, *feat.shape[1:]) + else: + feat_style = feat_style.repeat(1, b // 2, 1, 1, 1) + feat_style = torch.cat([feat_style[:, :1], scale * feat_style[:, 1:]], dim=1) + return feat_style.reshape(*feat.shape) + + +def concat_first(feat: T, dim=2, scale=1.) -> T: + feat_style = expand_first(feat, scale=scale) + return torch.cat((feat, feat_style), dim=dim) + + +def calc_mean_std(feat, eps: float = 1e-5) -> tuple[T, T]: + feat_std = (feat.var(dim=-2, keepdims=True) + eps).sqrt() + feat_mean = feat.mean(dim=-2, keepdims=True) + return feat_mean, feat_std + + +def adain(feat: T) -> T: + feat_mean, feat_std = calc_mean_std(feat) + feat_style_mean = expand_first(feat_mean) + feat_style_std = expand_first(feat_std) + feat = (feat - feat_mean) / feat_std + feat = feat * feat_style_std + feat_style_mean + return feat + + +class DefaultAttentionProcessor(nn.Module): + + def __init__(self): + super().__init__() + self.processor = attention_processor.AttnProcessor2_0() + + def __call__(self, attn: attention_processor.Attention, hidden_states, encoder_hidden_states=None, + attention_mask=None, **kwargs): + return self.processor(attn, hidden_states, encoder_hidden_states, attention_mask) + + +class SharedAttentionProcessor(DefaultAttentionProcessor): + + def shifted_scaled_dot_product_attention(self, attn: attention_processor.Attention, query: T, key: T, value: T) -> T: + logits = torch.einsum('bhqd,bhkd->bhqk', query, key) * attn.scale + logits[:, :, :, query.shape[2]:] += self.shared_score_shift + probs = logits.softmax(-1) + return torch.einsum('bhqk,bhkd->bhqd', probs, value) + + def shared_call( # pylint: disable=unused-argument + self, + attn: attention_processor.Attention, + hidden_states, + encoder_hidden_states=None, + attention_mask=None, + **kwargs + ): + + residual = hidden_states + input_ndim = hidden_states.ndim + if input_ndim == 4: + batch_size, channel, height, width = hidden_states.shape + hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2) + batch_size, sequence_length, _ = ( + hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape + ) + + if attention_mask is not None: + attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size) + # scaled_dot_product_attention expects attention_mask shape to be + # (batch, heads, source_length, target_length) + attention_mask = attention_mask.view(batch_size, attn.heads, -1, attention_mask.shape[-1]) + + if attn.group_norm is not None: + hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2) + + query = attn.to_q(hidden_states) + key = attn.to_k(hidden_states) + value = attn.to_v(hidden_states) + inner_dim = key.shape[-1] + head_dim = inner_dim // attn.heads + + query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2) + key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2) + value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2) + # if self.step >= self.start_inject: + if self.adain_queries: + query = adain(query) + if self.adain_keys: + key = adain(key) + if self.adain_values: + value = adain(value) + if self.share_attention: + key = concat_first(key, -2, scale=self.shared_score_scale) + value = concat_first(value, -2) + if self.shared_score_shift != 0: + hidden_states = self.shifted_scaled_dot_product_attention(attn, query, key, value,) + else: + hidden_states = nnf.scaled_dot_product_attention( + query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False + ) + else: + hidden_states = nnf.scaled_dot_product_attention( + query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False + ) + # hidden_states = adain(hidden_states) + hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim) + hidden_states = hidden_states.to(query.dtype) + + # linear proj + hidden_states = attn.to_out[0](hidden_states) + # dropout + hidden_states = attn.to_out[1](hidden_states) + + if input_ndim == 4: + hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width) + + if attn.residual_connection: + hidden_states = hidden_states + residual + + hidden_states = hidden_states / attn.rescale_output_factor + return hidden_states + + def __call__(self, attn: attention_processor.Attention, hidden_states, encoder_hidden_states=None, + attention_mask=None, **kwargs): + if self.full_attention_share: + _b, n, _d = hidden_states.shape + hidden_states = einops.rearrange(hidden_states, '(k b) n d -> k (b n) d', k=2) + hidden_states = super().__call__(attn, hidden_states, encoder_hidden_states=encoder_hidden_states, + attention_mask=attention_mask, **kwargs) + hidden_states = einops.rearrange(hidden_states, 'k (b n) d -> (k b) n d', n=n) + else: + hidden_states = self.shared_call(attn, hidden_states, hidden_states, attention_mask, **kwargs) + + return hidden_states + + def __init__(self, style_aligned_args: StyleAlignedArgs): + super().__init__() + self.share_attention = style_aligned_args.share_attention + self.adain_queries = style_aligned_args.adain_queries + self.adain_keys = style_aligned_args.adain_keys + self.adain_values = style_aligned_args.adain_values + self.full_attention_share = style_aligned_args.full_attention_share + self.shared_score_scale = style_aligned_args.shared_score_scale + self.shared_score_shift = style_aligned_args.shared_score_shift + + +def _get_switch_vec(total_num_layers, level): + if level <= 0: + return torch.zeros(total_num_layers, dtype=torch.bool) + if level >= 1: + return torch.ones(total_num_layers, dtype=torch.bool) + to_flip = level > .5 + if to_flip: + level = 1 - level + num_switch = int(level * total_num_layers) + vec = torch.arange(total_num_layers) + vec = vec % (total_num_layers // num_switch) + vec = vec == 0 + if to_flip: + vec = ~vec + return vec + + +def init_attention_processors(pipeline: StableDiffusionXLPipeline, style_aligned_args: StyleAlignedArgs | None = None): + attn_procs = {} + unet = pipeline.unet + number_of_self, number_of_cross = 0, 0 + num_self_layers = len([name for name in unet.attn_processors.keys() if 'attn1' in name]) + if style_aligned_args is None: + only_self_vec = _get_switch_vec(num_self_layers, 1) + else: + only_self_vec = _get_switch_vec(num_self_layers, style_aligned_args.only_self_level) + for i, name in enumerate(unet.attn_processors.keys()): + is_self_attention = 'attn1' in name + if is_self_attention: + number_of_self += 1 + if style_aligned_args is None or only_self_vec[i // 2]: + attn_procs[name] = DefaultAttentionProcessor() + else: + attn_procs[name] = SharedAttentionProcessor(style_aligned_args) + else: + number_of_cross += 1 + attn_procs[name] = DefaultAttentionProcessor() + + unet.set_attn_processor(attn_procs) + + +def register_shared_norm(pipeline: StableDiffusionXLPipeline, + share_group_norm: bool = True, + share_layer_norm: bool = True, + ): + def register_norm_forward(norm_layer: nn.GroupNorm | nn.LayerNorm) -> nn.GroupNorm | nn.LayerNorm: + if not hasattr(norm_layer, 'orig_forward'): + setattr(norm_layer, 'orig_forward', norm_layer.forward) # noqa + orig_forward = norm_layer.orig_forward + + def forward_(hidden_states: T) -> T: + n = hidden_states.shape[-2] + hidden_states = concat_first(hidden_states, dim=-2) + hidden_states = orig_forward(hidden_states) + return hidden_states[..., :n, :] + + norm_layer.forward = forward_ + return norm_layer + + def get_norm_layers(pipeline_, norm_layers_: dict[str, list[nn.GroupNorm | nn.LayerNorm]]): + if isinstance(pipeline_, nn.LayerNorm) and share_layer_norm: + norm_layers_['layer'].append(pipeline_) + if isinstance(pipeline_, nn.GroupNorm) and share_group_norm: + norm_layers_['group'].append(pipeline_) + else: + for layer in pipeline_.children(): + get_norm_layers(layer, norm_layers_) + + norm_layers = {'group': [], 'layer': []} + get_norm_layers(pipeline.unet, norm_layers) + return [register_norm_forward(layer) for layer in norm_layers['group']] + [register_norm_forward(layer) for layer in + norm_layers['layer']] + + +class Handler: + + def register(self, style_aligned_args: StyleAlignedArgs): + self.norm_layers = register_shared_norm(self.pipeline, style_aligned_args.share_group_norm, + style_aligned_args.share_layer_norm) + init_attention_processors(self.pipeline, style_aligned_args) + + def remove(self): + for layer in self.norm_layers: + layer.forward = layer.orig_forward + self.norm_layers = [] + init_attention_processors(self.pipeline, None) + + def __init__(self, pipeline: StableDiffusionXLPipeline): + self.pipeline = pipeline + self.norm_layers = [] diff --git a/modules/styles.py b/modules/styles.py index 0dd48eb7f..d0228d33a 100644 --- a/modules/styles.py +++ b/modules/styles.py @@ -52,7 +52,7 @@ def check_files(prompt, wildcard, files): choice = random.choice(lines).strip(' \n') if '|' in choice: choice = random.choice(choice.split('|')).strip(' []{}\n') - prompt = prompt.replace(f"__{wildcard}__", choice) + prompt = prompt.replace(f"__{wildcard}__", choice, 1) shared.log.debug(f'Wildcards apply: wildcard="{wildcard}" choice="{choice}" file="{file}" choices={len(lines)}') replaced.append(wildcard) return prompt, True diff --git a/modules/textual_inversion/textual_inversion.py b/modules/textual_inversion/textual_inversion.py index dd4203a4f..de12021a5 100644 --- a/modules/textual_inversion/textual_inversion.py +++ b/modules/textual_inversion/textual_inversion.py @@ -4,13 +4,14 @@ import torch import safetensors.torch from PIL import Image -from modules import shared, devices, sd_models, errors +from modules import shared, devices, errors from modules.textual_inversion.image_embedding import embedding_from_b64, extract_image_data_embed from modules.files_cache import directory_files, directory_mtime, extension_filter debug = shared.log.trace if os.environ.get('SD_TI_DEBUG', None) is not None else lambda *args, **kwargs: None debug('Trace: TEXTUAL INVERSION') +supported_models = ['ldm', 'sd', 'sdxl'] def list_embeddings(*dirs): @@ -274,6 +275,8 @@ def load_diffusers_embedding(self, filename: Union[str, List[str]] = None, data: overwrite = bool(data) if not shared.sd_loaded: return + if not shared.opts.diffusers_enable_embed: + return embeddings, skipped = open_embeddings(filename) or convert_bundled(data) for skip in skipped: self.skipped_embeddings[skip.name] = skipped @@ -368,7 +371,7 @@ def load_from_file(self, path, filename): self.skipped_embeddings[name] = embedding def load_from_dir(self, embdir): - if sd_models.model_data.sd_model is None: + if not shared.sd_loaded: shared.log.info('Skipping embeddings load: model not loaded') return if not os.path.isdir(embdir.path): @@ -388,6 +391,8 @@ def load_from_dir(self, embdir): def load_textual_inversion_embeddings(self, force_reload=False): if not shared.sd_loaded: return + if shared.sd_model_type not in supported_models: + return t0 = time.time() if not force_reload: need_reload = False diff --git a/modules/timer.py b/modules/timer.py index 8a5db726d..43e859140 100644 --- a/modules/timer.py +++ b/modules/timer.py @@ -7,6 +7,7 @@ def __init__(self): self.start = time.time() self.records = {} self.total = 0 + self.profile = False def elapsed(self, reset=True): end = time.time() @@ -15,17 +16,24 @@ def elapsed(self, reset=True): self.start = end return res + def add(self, name, t): + if name not in self.records: + self.records[name] = t + else: + self.records[name] += t + def record(self, category=None, extra_time=0, reset=True): e = self.elapsed(reset) if category is None: category = sys._getframe(1).f_code.co_name # pylint: disable=protected-access if category not in self.records: self.records[category] = 0 - self.records[category] += e + extra_time self.total += e + extra_time def summary(self, min_time=0.05, total=True): + if self.profile: + min_time = -1 res = f"{self.total:.2f} " if total else '' additions = [x for x in self.records.items() if x[1] >= min_time] if not additions: @@ -34,6 +42,8 @@ def summary(self, min_time=0.05, total=True): return res def dct(self, min_time=0.05): + if self.profile: + return {k: round(v, 4) for k, v in self.records.items()} return {k: round(v, 2) for k, v in self.records.items() if v >= min_time} def reset(self): diff --git a/modules/txt2img.py b/modules/txt2img.py index 2f0e2f4b3..e82c744a2 100644 --- a/modules/txt2img.py +++ b/modules/txt2img.py @@ -88,7 +88,7 @@ def txt2img(id_task, state, p.scripts = scripts.scripts_txt2img p.script_args = args p.state = state - processed = scripts.scripts_txt2img.run(p, *args) + processed: processing.Processed = scripts.scripts_txt2img.run(p, *args) if processed is None: processed = processing.process_images(p) processed = scripts.scripts_txt2img.after(p, processed, *args) diff --git a/modules/ui_common.py b/modules/ui_common.py index 9c4bb5cdc..3e7c68bec 100644 --- a/modules/ui_common.py +++ b/modules/ui_common.py @@ -245,10 +245,18 @@ def create_output_panel(tabname, preview=True, prompt=None, height=None): gr.HTML(value="", elem_id="main_info", visible=False, elem_classes=["main-info"]) # columns are for <576px, <768px, <992px, <1200px, <1400px, >1400px result_gallery = gr.Gallery(value=[], - label='Output', show_label=False, show_download_button=True, allow_preview=True, container=False, preview=preview, - columns=4, object_fit='scale-down', height=height, + label='Output', + show_label=False, + show_download_button=True, + allow_preview=True, + container=False, + preview=preview, + columns=4, + object_fit='scale-down', + height=height, elem_id=f"{tabname}_gallery", - ) + elem_classes=["gallery_main"], + ) if prompt is not None: interrogate_clip_btn, interrogate_booru_btn = ui_sections.create_interrogate_buttons('control') interrogate_clip_btn.click(fn=interrogate_clip, inputs=[result_gallery], outputs=[prompt]) diff --git a/modules/ui_control.py b/modules/ui_control.py index 0bf070036..cd78313a7 100644 --- a/modules/ui_control.py +++ b/modules/ui_control.py @@ -9,11 +9,11 @@ from modules.control.units import lite # vislearn ControlNet-XS from modules.control.units import t2iadapter # TencentARC T2I-Adapter from modules.control.units import reference # reference pipeline -from modules import errors, shared, progress, ui_components, ui_symbols, ui_common, ui_sections, generation_parameters_copypaste, call_queue, scripts, masking, images, processing_vae # pylint: disable=ungrouped-imports +from modules import errors, shared, progress, ui_components, ui_symbols, ui_common, ui_sections, generation_parameters_copypaste, call_queue, scripts, masking, images, processing_vae, timer # pylint: disable=ungrouped-imports from modules import ui_control_helpers as helpers -gr_height = None +gr_height = 512 max_units = shared.opts.control_max_units units: list[unit.Unit] = [] # main state variable controls: list[gr.component] = [] # list of gr controls @@ -21,13 +21,41 @@ debug('Trace: CONTROL') -def return_controls(res): +def return_stats(t: float = None): + if t is None: + elapsed_text = '' + else: + elapsed = time.perf_counter() - t + elapsed_m = int(elapsed // 60) + elapsed_s = elapsed % 60 + elapsed_text = f"Time: {elapsed_m}m {elapsed_s:.2f}s |" if elapsed_m > 0 else f"Time: {elapsed_s:.2f}s |" + summary = timer.process.summary(min_time=0.25, total=False).replace('=', ' ') + gpu = '' + cpu = '' + if not shared.mem_mon.disabled: + vram = {k: -(v//-(1024*1024)) for k, v in shared.mem_mon.read().items()} + peak = max(vram['active_peak'], vram['reserved_peak'], vram['used']) + used = round(100.0 * peak / vram['total']) if vram['total'] > 0 else 0 + if used > 0: + gpu += f"| GPU {peak} MB {used}%" + gpu += f" | retries {vram['retries']} oom {vram['oom']}" if vram.get('retries', 0) > 0 or vram.get('oom', 0) > 0 else '' + ram = shared.ram_stats() + if ram['used'] > 0: + cpu += f"| RAM {ram['used']} GB {round(100.0 * ram['used'] / ram['total'])}%" + return f"Time: {elapsed_text} | {summary} {gpu} {cpu}