Setting up Stable Diffusion on MacOS for generating QR codes and other experiments

8 min readJun 26, 2023

AI-generated QR codes have been a trending hot topic recently, and I wanted to create a few on my own. Below is a summary of the various articles and resources I found on the web about installing the necessary packages and models, generating the QR codes, and configuring Stable Diffusion and ControlNet to get the intended results.

This is meant as a quick getting started guide. For more in-depth information, consult the references at the end of the article and the related packages’ documentation.

This worked for me on both Intel and Apple Silicon-based Macs (and likely a similar procedure should succeed on Windows or Linux). Alternatively, you can also use a paid cloud Stable Diffusion service such as RunDiffusion.

The steps are essentially as follows:

Installing Stable Diffusion and models
Installing the ControlNet extension and models
Generating QR codes
Configuring Stable Diffusion / ControlNet to generate the image

Note that Stable Diffusion and ControlNet are working fine overall, but I’ve had limited success so far in generating functional QR code images. It requires skills and patience.

With 16Gb of RAM, 512x512 or 640x360 resolution appear to be the limit, and it’s not uncommon for my 2017 Intel iMac Pro to overheat and shut down. So I’m considering configuring an Amazon EC2 instance for better performance and stability. But this is an effective way to get started and learn.

Finally, AI being AI, within a few days of drafting this article, several services have emerged to create QR, such as Quickqr.art or QR Craft. If you just want to create QR without having to set up your own stable diffusion environment, you may want to try some of these instead.

Setup Stable Diffusion

We will be using the popular AUTOMATIC1111 web UI to run Stable Diffusion.

1. Make sure you have Brew installed on your Mac

/bin/bash -c “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"```

2. Install the following packages

brew install cmake protobuf rust git wget python@3.10

3. Checkout AUTOMATIC1111 stable-diffusion-webui from GiT

Make sure you use a location with lots of disk space available (like 50+Gb), as models will be stored under that project.

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui

4. Download models for Stable Diffusion

The files need to be saved under the stable-diffusion-webui/models/Stable-diffution directory. There are several you can potentially install, here are the ones I have used so far:

v1–5-pruned-emaonly.ckpt [4.27Gb] (ema-only weight version that uses less VRAM — suitable for inference)
GhostMix [2.24Gb] (recommended in one of the referenced articles)
DreamShaper [1.99Gb]

5. Run and test Stable Diffusion

At this point, you should be able to run Stable Diffusion by executing ./webui.sh shell command from the stable-diffusion-webui directory.

Note that on M1 Mac, I had to use ./webui.sh --no-half with Python 3.10. to circumvent a RuntimeError: “upsample_nearest2d_channels_last” not implemented for ‘Half' error.

This will start the server, and you can then navigate to the following URL in your browser: http://localhost:7860/. You should see the UI below, and you can generate a test image by entering a test prompt in txt2img tab.

This takes a little while to start the first time as the Python virtual environment is created and relevant packages and other models are installed.

Progress can be tracked in both the command shell and the UI. Note that resolution may be constrained by your system memory. I quickly get a RuntimeError: MPS backend out of memory on my 16Gb system.

Setup ControlNet

ControlNet is an extension of Stable Diffusion that enables fine-grained control over image generation, enabling more precise and flexible control over Stable Diffusion’s generation process.

1. Install in Web UI

You can add ControlNet from the web UI extension tab by going to Install from URL tab and pasting in the following link: https://github.com/Mikubill/sd-webui-controlnet, which will deploy it under webui\extensions\sd-webui-controlnet. Restart the server to activate.

You now should have a new ControlNet section in the UI.

Note that we will be using more than one control net unit. If you go to Settings under ControlNet, you will see that the Multi ControlNet: Max models amount is set to 3 (which is good). But for some reason, this is not reflected in the UI right away. To fix this, click Apply Settings and then Reload UI.

2. Download and install ControlNet models

ControlNet needs its own models, which can be retrieved from the Hugging Face repository. Visit that page, download all files with the .pth, and save them in the stable-diffusion-webui\extensions\sd-webui-controlnet\models directory. Note that the corresponding .yaml files are already present. This may take a while, depending on your connection speed (1.45Gb/model).

3. Download and install additional ControlNet models

in addition to the above, you can also install the T2IA models from https://civitai.com/models/17220 (in the same directory).

Additional models can be found on Hugging Faces and other repositories. These can be stored in the stable-diffusion-webui/models/ControlNet directory. The following are of particular interest (used below):

Generating QR codes

You naturally will need QR codes .png files to use as inputs with Stable Diffusion. Free sites to generate QR codes are all over the web. What is important is to make sure you use one that allows you to select error correction and always set this option to high. For example:

My initial QR attempts

Various techniques can be found on the web and social networks to produce AI-generated QR codes. Below are the ones I have used so far, with mixed results. This seems to require quite a bit of learning, patience in tuning the various models’ parameters, potentially some image post-processing, and a bit of luck.

But at least my environment is up and running, which is what I wanted to achieve and share.

img2img

The first technique I tried is described in this article from KingBarnaby and uses the img2img generative feature. This suggests optionally using LoRA models to achieve specific styles. You can find more about setting up and using LoRA in this guide.

In this approach, we essentially use the QR image as input to both Stable Diffusion and ControlNet with the following basic settings:

LoRA model weights: 0.8 Sampler: DPM++ 2M Karras Steps: 80–100 CFG: 7 Preprocessor: tile_resample ControlNet Model: control_v11f1e_sd15_tile (link above) ControlNet Weight: 0.85–1.25 Denoising Strength: 1 Pixel Perfect: Enabled Starting: 0.35–0.4 Ending: 0.75–0.9 Control Mode: Balanced

While the results are visually appealing, I failed to generate a QR code that works. As the author mentions, finding the right settings can take time.

The article is nonetheless very insightful and provides good documentation on QR codes. It also suggests post-processing the generated image with your favorite editors like PhotoPea or Photoshop.

So while this did not work for me, I learned a lot from this exercise. It is likely that great outputs can be generated, but I moved on to explore other options.

img2txt with inpaint_global

The second approach I tried is described by Rowan Cheung in this Twitter post, which received lots of attention and went viral.

This technique uses img2txt and two ContolNet units, both using the inpaint_global_harmonious preprocessor and the QR code as input.

For the first ControlNet0 unit, use the “brightness” model with a Control Weight at 0.35.
For the first ControlNet1 unit, use the “tile” model with a Control Weight at 0.65, Starting Control Step at 0.35, and Ending Control Step at 0.75.

This is nicely illustrated in the short videos available in the original tweet.

Finding the right prompt and tweaking the settings important in producing a functional QR code.

One issue is that I could not locate the control_v1p_sd15_brightness ControlNet model he recommends using. This seems to be for Stable Diffusion v1, while we are now using v1.5. So instead used the inpaint_global, which is probably not ideal.

Here is one example I was able to reproduce:

Using QR fine-tuned models

The third approach I explored is the use ControlNet model fine-tuned for QR code generations that started to emerge. I tried the following two (previously mentioned above in the installation steps)

This makes things a bit easier, as the only thing you need to do is to select the ControlNet model, use your QR code as input, create a prompt, and generate. Then increase the ControlNet model weight or start/end points until it succeeds. See the models’ documentation for instructions and examples.

Note that you will have to generate many images to succeed. Use the Stable Diffusion batch option to create sets of 5–10 images and try them with your favorite QR reader.

Online services

AI being AI, things evolve very quickly. As I’m about to publish this article, a few online services have become available to generate QR codes, such as Discord-based Quickqr.art server or the QR Craft website.

Some crazier stuff is also coming out, like augmented reality QR codes by Ian Curtis.

So who knows what will be out there by the time you read this?

I nonetheless found this entire exercise of installing locally very insightful and educational. Hope this article helps and inspires you to explore the Stable Diffusion universe. See the references below for more info.

Other experiments

Since I had Stable Diffusion and ControlNet up and running, I also played with traditional generative techniques, in particular leveraging the ControlNet Canny preprocessor and model, using my company logo and illustrations to generate interesting images that could be used for marketing and other purposes. Here are some of my favorites.