models runwayml stable diffusion inpainting
runwayml/stable-diffusion-inpainting is a versatile text-to-image model capable of producing realistic images from text input and performing inpainting using masks. It was initialized with Stable-Diffusion-v-1-2 weights and underwent two training phases: 595k steps of regular training and 440k steps of inpainting training at 512x512 resolution using the "laion-aesthetics v2 5+" dataset. To enhance performance, 10% of text-conditioning was dropped during inpainting training. The model employed a UNet with 5 extra input channels, and synthetic masks were generated during training, with 25% of the data being masked for inpainting refinement. The model has various applications in research, art, education, and creative tools. However, there are strict guidelines for the model's use to prevent misuse and malicious activities. It should not be used to create harmful, offensive, or discriminatory content. Additionally, the model has limitations, such as difficulties with photorealism, rendering legible text, and generating complex compositions. The model's training data primarily containing English descriptions, which can lead to biases and limitations in generating non-English content. To enhance safety, a Safety Checker is recommended for use with this model.
The above summary was generated using ChatGPT. Review the original-model-card to understand the data used to train the model, evaluation metrics, license, intended uses, limitations and bias before using the model.
| Inference type | Python sample (Notebook) | CLI with YAML |
|---|---|---|
| Real time | text-to-image-inpainting-online-endpoint.ipynb | text-to-image-inpainting-online-endpoint.sh |
| Batch | text-to-image-inpainting-batch-endpoint.ipynb | text-to-image-inpainting-batch-endpoint.sh |
Inference with Azure AI Content Safety (AACS) samples
| Inference type | Python sample (Notebook) |
|---|---|
| Real time | safe-text-to-image-inpainting-online-deployment.ipynb |
| Batch | safe-text-to-image-inpainting-batch-endpoint.ipynb |
{
"input_data": {
"columns": ["prompt", "image", "mask"],
"data": [
{
"prompt": "Face of a yellow cat, high resolution, sitting on a park bench",
"image": "image1",
"mask_image": "mask1"
},
{
"prompt": "Face of a green cat, high resolution, sitting on a park bench",
"image": "image2",
"mask_image": "mask2"
}
],
"index": [0, 1]
}
}Note:
- "image1" and "image2" strings are base64 format.
- "mask1" and "mask2" strings are base64 format.
[
{
"prompt": "Face of a yellow cat, high resolution, sitting on a park bench",
"generated_image": "inpainted_image1",
"nsfw_content_detected": False
},
{
"prompt": "Face of a green cat, high resolution, sitting on a park bench",
"generated_image": "inpainted_image2",
"nsfw_content_detected": False
}
]Note:
- "inpainted_image1" and "inpainted_image2" strings are base64 format.
- If "nsfw_content_detected" is True then generated image will be totally black.
Model inference: visualization for the prompt - "a small flower vase featuring a blend of yellow and orange"
/output_gridviz_runwayml_stable_diffusion_inpainting.png)
Version: 5
Preview SharedComputeCapacityEnabled license : creativeml-openrail-m task : text-to-image training_dataset : LAION aesthetics v2.5+ author : Runway huggingface_model_id : runwayml/stable-diffusion-inpainting inference_compute_allow_list : ['Standard_NC6s_v3', 'Standard_NC12s_v3', 'Standard_NC24s_v3', 'Standard_NC24ads_A100_v4', 'Standard_NC48ads_A100_v4', 'Standard_NC96ads_A100_v4', 'Standard_ND40rs_v2', 'Standard_ND96amsr_A100_v4', 'Standard_ND96asr_v4']
View in Studio: https://ml.azure.com/registries/azureml/models/runwayml-stable-diffusion-inpainting/version/5
License: creativeml-openrail-m
SHA: 51388a731f57604945fddd703ecb5c50e8e7b49d
inference-min-sku-spec: 6|1|112|336
inference-recommended-sku: Standard_NC6s_v3, Standard_NC12s_v3, Standard_NC24s_v3, Standard_NC24ads_A100_v4, Standard_NC48ads_A100_v4,Standard_NC96ads_A100_v4, Standard_ND40rs_v2, Standard_ND96amsr_A100_v4, Standard_ND96asr_v4