Image editing models

1 · What it is

InstructPix2Pix takes an input image and a written instruction, then follows that instruction to edit the image. The authors train this conditional diffusion model on generated examples. At inference it generalizes to real images and user-written instructions. They base the editor on Stable Diffusion. The diffusion runs in the latent space of a pretrained variational autoencoder. That autoencoder has an encoder and a decoder.

Extra channels on the first convolutional layer concatenate the noisy latent with the encoded source. Weights on those new channels start at zero. The instruction reuses the text path that was built for captions. Two guidance scales trade off a match to the input image against a match to the instruction. A finetuned GPT-3 writes the instructions and the edited captions. The authors fine-tuned GPT-3 Davinci for one epoch with its default parameters. Prompt-to-Prompt keeps the paired generations similar. The resulting set has over 450,000 examples. The edit does not need per-example fine-tuning or inversion. It edits an image in a matter of seconds.

Imagen Editor is given an image, a masked area, and a text prompt, and it fills that masked area. It concatenates that image and mask with the diffusion latents. SDEdit adds noise to a guide drawn in RGB pixels, then denoises through a stochastic differential equation prior so the picture looks more realistic. It does not need task-specific training or inversion. Prompt-to-Prompt injects cross-attention maps during diffusion. Those maps tie the picture layout to each word in the prompt. Editing a real photo this way uses an inversion process. Emu Edit trains one model on sixteen editing tasks. A learned task embedding steers the model toward the right kind of edit. One model trained on all of the tasks beat a separate expert for each task.