FINE-TUNING AND CONTROLLING DIFFUSION MODELS

Systems and methods for fine-tuning diffusion models are described. Embodiments of the present disclosure obtain an input text indicating an element to be included in an image; generate a synthetic image depicting the element based on the input text using a diffusion model trained by comparing synthetic images depicting the element to training images depicting elements similar to the element and updating selected parameters corresponding to an attention layer of the diffusion model based on the comparison.