ControlNet Auxiliary Models

Common errors

• ImportError: cannot import name 'resize_image_with_pad' from 'controlnet_aux.util'

  cause This error typically occurs when `controlnet-aux` is either outdated, or there's a conflict with other installed packages (e.g., an older version of `diffusers` or related utilities) that expect a different internal structure or API.
  fix Update `controlnet-aux` to the latest version: `pip install -U controlnet-aux`. If the issue persists, try reinstalling it in a fresh virtual environment or checking for conflicting `diffusers` versions (`pip list | grep diffusers`).

• Nodes Not Appearing (in ComfyUI after installing controlnet_aux)

  cause When `controlnet-aux` is integrated as a custom node in environments like ComfyUI, this often indicates missing Python dependencies required by specific annotators or a general installation issue preventing the nodes from loading.
  fix Check the console log immediately after starting ComfyUI for specific error messages regarding missing packages. Manually install any reported missing dependencies (`pip install <package_name>`). Ensure all requirements from `requirements.txt` (if applicable for a custom node wrapper) are met.

• TypeError: expected size to be one of int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int], but got size with types [<class 'numpy.int64'>, <class 'numpy.int64'>]

  cause This type error usually happens in image processing steps where a function expecting an integer or tuple of integers for dimensions receives NumPy integers (`numpy.int64`) instead, which are not implicitly cast.
  fix Explicitly convert NumPy integers to standard Python integers using `int()` before passing them as arguments to functions that expect Python native types. For example, `(int(width), int(height))`.

Warnings

• gotcha DWPose can be significantly slower when running on CPU. For optimal performance, especially with DWPose or AnimalPose, consider using TorchScript checkpoints or ONNXRuntime with GPU acceleration.
  Fix: Install `easy-dwpose` (which uses ONNX models) and ensure your environment is configured for ONNXRuntime or PyTorch with GPU (e.g., CUDA) for faster inference. Refer to documentation for specific setup instructions.
• gotcha When processing images for ControlNet, it is recommended that the input image resolution (or target detection resolution) be a multiple of 64 to maintain the aspect ratio and ensure optimal performance and output quality.
  Fix: Resize your input images or adjust `detect_resolution` and `image_resolution` parameters to be multiples of 64. PIL's `resize` method can be used for this.
• breaking Installing `controlnet-aux` alongside older `comfyui_controlnet_preprocessors` or similar custom nodes in ComfyUI can lead to import conflicts or unexpected behavior due to overlapping functionalities and potentially outdated dependencies.
  Fix: If using ComfyUI, ensure that `comfyui_controlnet_preprocessors` is completely removed before installing `controlnet-aux` based custom nodes. Always follow the installation instructions specific to your ComfyUI custom node setup.

Install

• pip install -U controlnet-aux Latest stable release
• pip install easy-dwpose For DWPose support (recommended)

Imports

• Processor

  from controlnet_aux.processor import Processor

  Recommended unified interface for most annotators
• HEDdetector

  from controlnet_aux import HEDdetector

  For direct import of individual detectors/annotators
• OpenposeDetector

  from controlnet_aux import OpenposeDetector

  For direct import of individual detectors/annotators

Quickstart

This quickstart demonstrates how to use the unified `Processor` class to load and apply a ControlNet annotator (e.g., OpenPose) to an input image. The `Processor` dynamically loads the necessary models based on the `processor_id` you specify.

import requests
from PIL import Image
from io import BytesIO
from controlnet_aux.processor import Processor

# Load an image from a URL
url = "https://huggingface.co/lllyasviel/sd-controlnet-openpose/resolve/main/images/pose.png"
response = requests.get(url)
img = Image.open(BytesIO(response.content)).convert("RGB")

# Instantiate the processor for a specific ControlNet annotator (e.g., 'openpose')
# Common processor_ids include: 'canny', 'depth_leres', 'openpose', 'scribble_hed', 'lineart_realistic', 'dwpose'
processor_id = 'openpose'
processor = Processor(processor_id)

# Process the image
processed_image = processor(img, to_pil=True)

# Display or save the processed image
# processed_image.show() # Uncomment to display
print(f"Image processed successfully using {processor_id}. Output image size: {processed_image.size}")