Segmentation - What's great / frustrating

[svenjuga]

Hello mcmonkey,

first of all, thank you for your great work.

I come from Auto1111 and ForgeUI, have been experimenting with SwarmUI almost daily for about 2 months now and find SwarmUI a very useful user interface that is fast and stable.

I like a lot of things, but some aspects are catastrophic from a user's perspective and lead to frustration, especially because there's no proper guide for it. While it's helpful that each function has a small explanation (behind the question mark on the side), most explanations are incomprehensible to me.

Like most users, I'm not a python expert. Therefore, it took days to understand how Segmentation works. Countless attempts with yoloV8 models and tinkering with the values. It's great that you can also optimize a second face with it, but it took forever to figure out how to do that. And I haven't gotten any further with segmenting so far. You say: "segment:AnyTextHere", but that doesn't work. How do I approach it when I want to improve specific areas like hair, legs, nose, hands, ...? This requires detailed instructions and links to the usable models. For me it's just impossible to figure out.

Now I've read an issue mentioned that after "segment:text..." you should enter the entire prompt plus the area to be segmented. No idea if that works, but why? It's unnecessarily complicated. If you want to segment multiple areas, the prompt would become unmanageably long.

I can't get along with "object" and "region" at all. It only produces garbage or, in most cases, no change at all. How exactly does it work? Please also provide examples. It could be helpful, I think.

Please keep in mind that most users are neither programmers nor python experts. The majority wants to turn ideas into images and not pursue a degree in computer science. Spending hours searching in various forums is absolutely frustrating.

Do you know why Auto1111 and ForgeUI are so popular? Because they are easy to handle. SwarmUI is a great user interface, but it's still too complicated.

My tips and wishes (what's urgently needed):

1. Please expand your "Basic Usage" guide to include all basic functions. All functions of SwarmUI should at least be described there and provided with brief examples. Especially: Masking, Regional Prompting, Segmentation, Img2Img/Edit/Inpaint etc.

2. If possible, incorporate the segmentation (possibly also the "object" and "region" functions) into the dropdown menu on the left side. Yes, there's already something there, but I don't understand it. It should be simply structured and understandable, like Refiner and ControlNet are - big praise for that by the way!

3. Really great would be extensions like Adetailer (for segmentation) or something like ForgeCouple (for regional prompting) or similarly structured functions in the dropdown.

Thanks again for your great work. I would recommend SwarmUI to everyone if it were easier to handle, or if the usage of the above-mentioned functions were clearly explained in the "Basic Usage" guide of the Readme.

Best regards,
svenjuga

[mcmonkey4eva]

You says: segment:AnyTextHere, but that doesn't work. How do I approach it when I want to improve specific areas like hair, legs, nose, hands, ...? This requires detailed instructions and links to the usable models. For me it's just impossible to figure out.
You don't need models. You just <segment:legs> really nice detailed legs

Docs for segmentation are here https://github.com/mcmonkeyprojects/SwarmUI/blob/master/docs/Features/Prompt%20Syntax.md#automatic-segmentation-and-refining

Also you can just type a < in the prompt box to get on-screen info popups, and after you type <segment: it shows you how you can use it:

Most issues with quality on segment come down to one of:

• (A) prompting: make sure you prompt it well! eg if you're using a pony model, you need that annoying score_ spam stuff or else it'll be ugly. You can also leave the prompt for a segment blank to use your global prompt (like adetailer does) albeit that's usually worse imo, a special prompt for the segment gets better results.
• (B) masking: generally you want more mask, not less. Eg if you want to change the eyes of a character, <segment:face> is often better than <segment:eyes> just because the eyes mask will be too small and confuse the model
• (B.2) creativity value: different creativities are useful for different changes. If you want to totally replace something, creativity 1 or 0.9 makes sense, whereas if you just want to very slightly refine the existing image, maybe creativity 0.3 is better. The default is 0.6, which is decent for most average cases, but customizing it to your specific case can be better.
• (C) model: you get what you use. If you use an ugly model to inpaint a segment you get ugly results. There's no magic here, it's just rerunning the model over a segment of the image.

---

For example re "yes you can literally just type legs in", here I used the prompt full body photo of a woman standing in the middle of a street <segment:legs,1,0.5> glowing orange oompa loompa legs

I also enabled two advanced parameters:

That generated this initial image:

and the saved segment mask from that was:

Which then regenerated:

.. So, yeah, literally <segment:legs> works to match the legs and inpaint them. In my example I did <segment:legs,1,0.5> wherein 0.5 is just leaving the threshold at default, and the 1 is to set an explicit creativity value raised to 1 because changing the fundamental coloring of an image needs high creativity (and I'm using Flux which requires basically maxing it out to have a chance)
(Note: the weird outline around her legs is a known thing that flux does, other models don't do that. This can be fixed with a reset-to-norm but eh not relevant here, just trying to show the segment logic)

---

Do you know why Auto1111 and ForgeUI are so popular? Because they are easy to handle. SwarmUI is a great user interface, but it's still too complicated.
Do you know why Swarm exists? ... Well several reasons, but one of them is that Auto1111 is way too complicated and designed for turbonerds deep into AI, and I wanted Swarm's interface to be understandable to normal people. Swarm's UI is designed to be something I can show to somebody who's never used AI in their life and they can still get up to speed, while simultaneously having power tools in easy reach so once you're advanced beyond that beginner stage Swarm remains the best UI available.

You're just preaching to the choir here, while simultaneously not realizing you yourself are well advanced beyond the stage of a normal user. Normal people don't know or care what a yolo model is, but would love to easily add detail to parts of an image, which is why segment by default just lets you type in something and it searches for what you want without needing any models or any other nonsense. Customizing yolo models is an advanced/poweruser feature.

Really great would be extensions like Adetailer (for segmentation)
adetailer is just segment if you only use yolo models and leave the prompt input blank.

If possible, incorporate the segmentation (possibly also the and functions) into the dropdown on the side
There are settings for it there under Advanced -> Regional Prompting

The reason the segments themselves aren't there (at least atm) is because it's a dynamic list of as many segments as you want, that can take on a few entirely unique structures (eg prompted vs model based)
Note I do have plans to revamp all of the prompt-input options to have a proper UI in the future

Please expand your "Basic Usage" guide to include all basic functions. All functions of SwarmUI should at least be described there and provided with brief examples. Especially: Masking, Regional Prompting, Segmentation, Img2Img/Edit/Inpaint etc.
I do want to expand that and write more thorough beginner's guides.

re regional prompting, that's intentionally undocumented as I want to build out a proper interface for it (atm it's non-obvious and I don't want to write guides for how to do something that shouldn't be so hard to do in the first place).
(See above comment re I plan to revamp prompt input options to have proper UIs)

There are video tutorials for Swarm available btw, such as this one https://www.youtube.com/watch?v=HKX8_F1Er_w (Cefurkan 90 minute zero-to-hero with swarm)

[svenjuga]

Many thanks for the detailed explanations. I will study and check them out.

Unfortunately, the editor did not recognize my terms "object" and "region", (maybe because I put them in brackets, I changed that).
If you could also provide examples of use for those terms I would be very happy.

[mcmonkey4eva]

Also has in-UI docs:

As noted above, there's intentionally no fuller docs about it until there's a proper interface. For now the best available is if you use the Edit Image button and the Select tool, there's a Make Region button to turn your selection on the image editor into a valid region in the prompt.

[svenjuga]

Maybe it's just me, but the in-UI docs hardly could help. I think I'm the practical guy who needs examples / use cases. I've already tinkered with "object" und "region", but it doesn't work. Do I use them to ad an object by inpainting a region of an excisting image?

Btw. if SwarmUI is supposed to be beginner-friendly, as you say, such compressed, abstract explanations are not very helpful, I think.
But please don't get me wrong, this is neither an attack nor a demand. I would just like to be able to use SwarmUI to its full extent. (After all, I've already figured out how wildcards and the random term work. I'm really proud of that ;-))

Thanks for your time and effort!

[mcmonkey4eva]

For the third time:

re regional prompting, that's intentionally undocumented as I want to build out a proper interface for it (atm it's non-obvious and I don't want to write guides for how to do something that shouldn't be so hard to do in the first place).
(See above comment re I plan to revamp prompt input options to have proper UIs)

As noted above, there's intentionally no fuller docs about it until there's a proper interface