ControlNet is an add-on network for image diffusion models that steers the picture with an extra guide image, such as an edge outline, depth map or human pose.
In plain words
A text prompt alone gives little control over layout or poses. ControlNet adds a second input: a structural guide image. It works by copying blocks of a pretrained diffusion model. One copy stays locked, so the original model is preserved, and the other copy is trained to follow the new condition. The two are joined by zero convolutions, layers that start at zero so training does not damage the original model. It was first tested with Stable Diffusion.
A quick example
Imagine you have a photo of a room and want the same layout in a watercolour style. You extract its edges, for example with OpenCV's Canny edge detector, then pass that outline to a ControlNet together with the prompt. The new image follows the lines of the original room while the prompt sets the style.
Where you'll meet it
- Keeping the pose or layout of a reference image.
- Stable Diffusion and other image models, including through Diffusers.
- Combining several controls, such as edges plus depth.
Related
Sources
- PaperAdding Conditional Control to Text-to-Image Diffusion Models, Zhang et al. (arXiv) · read 6 Oct 2026
- Docslllyasviel/ControlNet, Lvmin Zhang (GitHub) · read 6 Oct 2026
- DocsControlNet, Hugging Face Diffusers documentation · read 6 Oct 2026
Checked by WhatIsAI against the sources above · 6 Oct 2026. Spotted an error? Report it.