Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat]Add strength in flux_fill pipeline (denoising strength for fluxfill) #10603

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

Suprhimp
Copy link

What does this PR do?

allows the fluxfill pipeline to have a denoising strength (denoising strength).

Before submitting

Who can review?

@yiyixuxu and @asomoza

I use fluxfill pipeline to edit image or outpatin. Yes it works very well but I don't agree Flux Fill pipeline does not require strength as an input like regular inpainting pipelines this line in the docs.

When remove object in the image, We need denoising strength I guess. without denoising strength I can't get clean image that remove object that I want.
let me gives you an example I tested.

mask
original_image

this is mask and original image that I want to edit.

And this is the result with denoising strength (0.6 with revised pipeline) and none(default pipeline)

cbimage (51)
7fabf5e8-9d0d-47d8-af7b-0371b4f29f21

I did lama inpaint to remove more clearly in both case but output is different.

As you can see with denoising strength we can have more control with the image quality. So I think there is no reason to not use denoising strength in fluxfill pipeline

I changed the fluxfill pipeline code with reference sdxl inpaint pipeline.
So, I think there would be things that need to be fixed.

Thanks for reading :)

Copy link
Collaborator

@hlky hlky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left some comments. However, we should note that Flux Fill is intended to fill areas based on a text description and not for object removal.

self.mask_processor = VaeImageProcessor(
vae_scale_factor=self.vae_scale_factor * 2,
vae_latent_channels=latent_channels,
vae_latent_channels=self.vae.config.latent_channels,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to use latent_channels here not the config directly. This allows pipelines to be used without the component e.g. FluxFillPipeline(vae=None, ...).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know that, Thanks, I changed. :)

@@ -627,6 +659,8 @@ def disable_vae_tiling(self):
# Copied from diffusers.pipelines.flux.pipeline_flux.FluxPipeline.prepare_latents
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we copy from FluxImg2ImgPipeline.prepare_latents or FluxInpaintPipeline.prepare_latents?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure , That would be more clean. Thanks for the review :)

@@ -809,6 +866,10 @@ def __call__(
self._joint_attention_kwargs = joint_attention_kwargs
self._interrupt = False

original_image = image
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep :) Maby that came from sdxl inpaint pipeline, but it is not used in this pipeline

src/diffusers/pipelines/flux/pipeline_flux_fill.py Outdated Show resolved Hide resolved
src/diffusers/pipelines/flux/pipeline_flux_fill.py Outdated Show resolved Hide resolved
src/diffusers/pipelines/flux/pipeline_flux_fill.py Outdated Show resolved Hide resolved
@@ -855,13 +952,13 @@ def __call__(
if masked_image_latents is not None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above # 6. Prepare mask and masked image latents.

@Suprhimp
Copy link
Author

Thankyou for the fast and kind review, I'll revise it and test :)

@Suprhimp
Copy link
Author

Suprhimp commented Jan 17, 2025

I've left some comments. However, we should note that Flux Fill is intended to fill areas based on a text description and not for object removal.

Hmm, Maby I missed the intention than ;)
Although I think we can use fluxfill with more controllable, with denosing strength:)

@hlky Thanks for the review again, I changed the code that you've reviewed.

@asomoza
Copy link
Member

asomoza commented Jan 17, 2025

Hi, you're saying you used lama to remove the object before doing this? If that's what you did, you're just essentially using this pipeline as an img2img one and the model as just a refiner.

The original fill model was trained to do this without the need of changing the strength and this is from the original implementation, I'm not saying that what you're doing is wrong though, people used redux in a more different and creative way and it works alright but I want to understand your use case since I haven't encountered the need to lower the strength to make it work.

Did you try what you're doing here with just the regular Flux model and the img2img pipeline?

@Suprhimp
Copy link
Author

Suprhimp commented Jan 17, 2025

Did you try what you're doing here with just the regular Flux model and the img2img pipeline?

Thanks for your response :)

I tried with sdxl model with img2img model. But not flux image2image.
And yep I understand what you said, Maby I'm just using the fluxfill pipeline as an refiner.

As I use this pipeline an image edit pipeline, I just want to use the single pipeline as an object adder or outpainter but also object remover.

If I have to use my case with flux img2img pipeline I can do that, but as you know flux model is quite large.
So I'd like to use it as refiner but also object adder or outpainter etc.

If I can use fluxfill pipeline with strength parameter, I don't need any other pipeline :)

@asomoza

@hlky
Copy link
Collaborator

hlky commented Jan 17, 2025

@Suprhimp To confirm, you're using lama cleaner to remove the object first?

@Suprhimp
Copy link
Author

Suprhimp commented Jan 17, 2025

To confirm, you're using lama cleaner to remove the object first?

@hlky Yep I used lamacleaner before both case (my pipeline, and default pipeline)

@ukaprch
Copy link

ukaprch commented Jan 18, 2025

lama with the refiner is very good at removing objects. I thought the whole point of Flux fill was to replace or insert an object. Big difference there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants