Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

no image2Image support ? #2

Open
Serasul1984 opened this issue Apr 23, 2024 · 17 comments
Open

no image2Image support ? #2

Serasul1984 opened this issue Apr 23, 2024 · 17 comments

Comments

@Serasul1984
Copy link

Image2Image is not supported ???

@ShenZhang-Shin
Copy link
Collaborator

Please wait, we will soon support it.

@Bluebang
Copy link

+1

@radames
Copy link

radames commented Apr 26, 2024

this will be amazing! just testing with pure controlnet without the reference image is suboptimal

image

@fkjkey
Copy link

fkjkey commented Apr 29, 2024

Is controlnet only supported in the XL version?
I used version 1.5 and got it
File "/root/anaconda3/envs/stable_fast/lib/python3.10/site-packages/hidiffusion/hidiffusion.py", line 383, in call
self.check_inputs(
TypeError: StableDiffusionControlNetPipeline.check_inputs() takes from 4 to 13 positional arguments but 17 were given

@ShenZhang-Shin
Copy link
Collaborator

Yes, We now only support XL+controlnet.

@fkjkey
Copy link

fkjkey commented Apr 30, 2024

but I used XL and got it
NameError: name 'scale_lora_layers' is not defined @ShenZhang-Shin

@ShenZhang-Shin
Copy link
Collaborator

but I used XL and got it NameError: name 'scale_lora_layers' is not defined @ShenZhang-Shin

Do you run hidiffusion with diffusers==0.27.0 or diffusers==0.25.0 ? We have tested the two versions and it works fine.
scale_lora_layers belongs to diffusers. Other versions may meet incompatibility

@ShenZhang-Shin
Copy link
Collaborator

ShenZhang-Shin commented May 7, 2024

Now we support image2image, welcome to try it to generate impressive images !
Here is an example.
Given a blurry image, hidiffusion can output a 2K clear image.
lich_king
hidi_lich_5_17_seed_25_conditioning_scale_0 5_Warcraft 3  Arthas stands in a snowy landscape, clad in armor with a skull motif, holding a glowing

@radames
Copy link

radames commented May 9, 2024

thanks @ShenZhang-Shin , amazing, put it together as a Space demo here https://huggingface.co/spaces/radames/Enhance-This-HiDiffusion-SDXL using the latest controlnet SDXL https://huggingface.co/TheMistoAI/MistoLine

QuickTime.Player.mp4

@fkjkey
Copy link

fkjkey commented May 9, 2024

但是我使用了XL并得到了NameError:未定义名称“scale_lora_layers”

您是否使用 diffusers==0.27.0 或 diffusers==0.25.0 运行 hidiffusion ?我们已经测试了这两个版本,它运行良好。 scale_lora_layers属于扩散器。其他版本可能不兼容

I use XL-lightning, I wonder if that's the reason

@ShenZhang-Shin
Copy link
Collaborator

Please provide your code with XL-lightning, I will reproduce it and check the reason

@ShenZhang-Shin
Copy link
Collaborator

thanks @ShenZhang-Shin , amazing, put it together as a Space demo here https://huggingface.co/spaces/radames/Enhance-This-HiDiffusion-SDXL using the latest controlnet SDXL https://huggingface.co/TheMistoAI/MistoLine

QuickTime.Player.mp4

Nice work. However, I find that the output image is not as good as my code output, such as Lara. There might be some problems to fix.

@ShenZhang-Shin
Copy link
Collaborator

thanks @ShenZhang-Shin , amazing, put it together as a Space demo here https://huggingface.co/spaces/radames/Enhance-This-HiDiffusion-SDXL using the latest controlnet SDXL https://huggingface.co/TheMistoAI/MistoLine

QuickTime.Player.mp4

The lara output with the same prompt and guidance scale on my machine. The output is better, especially in the background.

prompt: photography of lara croft 8k high definition award winning
negative prompt: underexposed, poorly drawn hands, duplicate hands, bad limbs, overexposed, bad art, beginner, amateur, abstract, disfigured, deformed
strength: 0.99
controlnet_conditioning_scale: 0.5
guidance_scale: 8.5

72_seed_98_conditioning_scale_0 5_photography of lara croft 8k high definition award winning

@radames
Copy link

radames commented May 9, 2024

very cool! I'm using this new controlnet net model and a custom sobel operator to generate the canny , https://huggingface.co/TheMistoAI/MistoLine , it would produce different outputs for sure

@ShenZhang-Shin
Copy link
Collaborator

very cool! I'm using this new controlnet net model and a custom sobel operator to generate the canny , https://huggingface.co/TheMistoAI/MistoLine , it would produce different outputs for sure

Wow, A new controlnet that Controls Every Line! Thanks for sharing. I will be going to use it with hidiffusion.

@fkjkey
Copy link

fkjkey commented May 9, 2024

Please provide your code with XL-lightning, I will reproduce it and check the reason

class Xl_make():
  def __init__(self):
      self.controlnet_canny = ControlNetModel.from_pretrained(
          "/home/lora_test/model/canny_xl", use_safetensors=True, torch_dtype=torch.float16
      )
      self.pipe = StableDiffusionXLControlNetImg2ImgPipeline.from_pretrained(
          "/home/lora_test/model/dreamshaper_xl_lightning",
          torch_dtype=torch.float16,
          controlnet=self.controlnet_canny,
          use_safetensors=True,
      ).to('cuda')
      self.pipe.scheduler = DPMSolverMultistepScheduler.from_config(
        self.pipe.scheduler.config,
        algorithm_type="sde-dpmsolver++"
      )
      apply_hidiffusion(self.pipe)
  
  def make(self, image):
      width, height = image.size
      if width < 832 or height < 832:
         image = image.resize((width*2, height*2))
         width, height = image.size
      if width > 832 or height > 832:
         image = image.resize((int(832 / max(width, height) * width), int(832 / max(width, height) * height)))
      width, height = image.size
      image = image.resize((width, height))

      canny_img = np.array(image)
      low_threshold = 100 
      high_threshold = 200
      canny = cv2.Canny(canny_img, low_threshold, high_threshold)
      canny = Image.fromarray(canny)

      new_img = self.pipe(
         prompt='oil painting,Masterpiece,best quality,nature,in a meadow,lake,sky'
         negative_prompt='bad quality, worst quality, text, signature, watermark, extra limbs,blurred, watermark, signature, low contrast, low resolution',
         image=image,
         control_image=canny,
         guidance_scale=4,
         strength=0.6
         num_inference_steps=10
         controlnet_conditioning_scale=1.0,
         generator=torch.manual_seed(42),
         control_guidance_start=0.0,
         control_guidance_end=0.6,
         clip_skip=2,
      ).images[0]
      new_img.save('/home/sdmade_material/new_img/new1.jpg')

      new_img.save(f'/home/sdmade_material/new_img/new1.5.jpg')

@ShenZhang-Shin
Copy link
Collaborator

Please provide your code with XL-lightning, I will reproduce it and check the reason

class Xl_make():
  def __init__(self):
      self.controlnet_canny = ControlNetModel.from_pretrained(
          "/home/lora_test/model/canny_xl", use_safetensors=True, torch_dtype=torch.float16
      )
      self.pipe = StableDiffusionXLControlNetImg2ImgPipeline.from_pretrained(
          "/home/lora_test/model/dreamshaper_xl_lightning",
          torch_dtype=torch.float16,
          controlnet=self.controlnet_canny,
          use_safetensors=True,
      ).to('cuda')
      self.pipe.scheduler = DPMSolverMultistepScheduler.from_config(
        self.pipe.scheduler.config,
        algorithm_type="sde-dpmsolver++"
      )
      apply_hidiffusion(self.pipe)
  
  def make(self, image):
      width, height = image.size
      if width < 832 or height < 832:
         image = image.resize((width*2, height*2))
         width, height = image.size
      if width > 832 or height > 832:
         image = image.resize((int(832 / max(width, height) * width), int(832 / max(width, height) * height)))
      width, height = image.size
      image = image.resize((width, height))

      canny_img = np.array(image)
      low_threshold = 100 
      high_threshold = 200
      canny = cv2.Canny(canny_img, low_threshold, high_threshold)
      canny = Image.fromarray(canny)

      new_img = self.pipe(
         prompt='oil painting,Masterpiece,best quality,nature,in a meadow,lake,sky'
         negative_prompt='bad quality, worst quality, text, signature, watermark, extra limbs,blurred, watermark, signature, low contrast, low resolution',
         image=image,
         control_image=canny,
         guidance_scale=4,
         strength=0.6
         num_inference_steps=10
         controlnet_conditioning_scale=1.0,
         generator=torch.manual_seed(42),
         control_guidance_start=0.0,
         control_guidance_end=0.6,
         clip_skip=2,
      ).images[0]
      new_img.save('/home/sdmade_material/new_img/new1.jpg')

      new_img.save(f'/home/sdmade_material/new_img/new1.5.jpg')

I see, man. I will reproduce it and fix the problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants