{"id":7111,"library":"controlnet-aux","title":"ControlNet Auxiliary Models","description":"ControlNet Auxiliary Models (controlnet-aux) provides a PyPI installable package of lllyasviel's ControlNet Annotators. It offers various preprocessors (e.g., Canny, OpenPose, Midas) for generating hint images like edges, depth maps, and poses, which are used to guide image generation with ControlNet models. The library is currently at version 0.0.10, released on May 8, 2025, with an irregular release cadence primarily driven by upstream ControlNet updates.","status":"active","version":"0.0.10","language":"en","source_language":"en","source_url":"https://github.com/patrickvonplaten/controlnet_aux","tags":["computer-vision","machine-learning","deep-learning","controlnet","image-processing","huggingface"],"install":[{"cmd":"pip install -U controlnet-aux","lang":"bash","label":"Latest stable release"},{"cmd":"pip install easy-dwpose","lang":"bash","label":"For DWPose support (recommended)"}],"dependencies":[{"reason":"Required for array manipulation","package":"einops","optional":false},{"reason":"For resource locking","package":"filelock","optional":false},{"reason":"For model downloading and management","package":"huggingface_hub","optional":false},{"reason":"For package metadata","package":"importlib_metadata","optional":false},{"reason":"Fundamental package for numerical computation","package":"numpy","optional":false},{"reason":"Core computer vision functionalities","package":"opencv-python-headless","optional":false},{"reason":"Image processing library","package":"pillow","optional":false},{"reason":"Image processing algorithms","package":"scikit-image","optional":false},{"reason":"Scientific computing tools","package":"scipy","optional":false},{"reason":"PyTorch Image Models, used by some annotators","package":"timm","optional":false},{"reason":"Main deep learning framework","package":"torch","optional":false},{"reason":"Vision datasets, models and transforms for PyTorch","package":"torchvision","optional":false},{"reason":"Lightweight DWPose implementation without MMDetection/MMCV/MMPose; highly recommended for DWPose.","package":"easy-dwpose","optional":true}],"imports":[{"note":"Recommended unified interface for most annotators","symbol":"Processor","correct":"from controlnet_aux.processor import Processor"},{"note":"For direct import of individual detectors/annotators","symbol":"HEDdetector","correct":"from controlnet_aux import HEDdetector"},{"note":"For direct import of individual detectors/annotators","symbol":"OpenposeDetector","correct":"from controlnet_aux import OpenposeDetector"}],"quickstart":{"code":"import requests\nfrom PIL import Image\nfrom io import BytesIO\nfrom controlnet_aux.processor import Processor\n\n# Load an image from a URL\nurl = \"https://huggingface.co/lllyasviel/sd-controlnet-openpose/resolve/main/images/pose.png\"\nresponse = requests.get(url)\nimg = Image.open(BytesIO(response.content)).convert(\"RGB\")\n\n# Instantiate the processor for a specific ControlNet annotator (e.g., 'openpose')\n# Common processor_ids include: 'canny', 'depth_leres', 'openpose', 'scribble_hed', 'lineart_realistic', 'dwpose'\nprocessor_id = 'openpose'\nprocessor = Processor(processor_id)\n\n# Process the image\nprocessed_image = processor(img, to_pil=True)\n\n# Display or save the processed image\n# processed_image.show() # Uncomment to display\nprint(f\"Image processed successfully using {processor_id}. Output image size: {processed_image.size}\")","lang":"python","description":"This quickstart demonstrates how to use the unified `Processor` class to load and apply a ControlNet annotator (e.g., OpenPose) to an input image. The `Processor` dynamically loads the necessary models based on the `processor_id` you specify."},"warnings":[{"fix":"Install `easy-dwpose` (which uses ONNX models) and ensure your environment is configured for ONNXRuntime or PyTorch with GPU (e.g., CUDA) for faster inference. Refer to documentation for specific setup instructions.","message":"DWPose can be significantly slower when running on CPU. For optimal performance, especially with DWPose or AnimalPose, consider using TorchScript checkpoints or ONNXRuntime with GPU acceleration.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Resize your input images or adjust `detect_resolution` and `image_resolution` parameters to be multiples of 64. PIL's `resize` method can be used for this.","message":"When processing images for ControlNet, it is recommended that the input image resolution (or target detection resolution) be a multiple of 64 to maintain the aspect ratio and ensure optimal performance and output quality.","severity":"gotcha","affected_versions":"All versions"},{"fix":"If using ComfyUI, ensure that `comfyui_controlnet_preprocessors` is completely removed before installing `controlnet-aux` based custom nodes. Always follow the installation instructions specific to your ComfyUI custom node setup.","message":"Installing `controlnet-aux` alongside older `comfyui_controlnet_preprocessors` or similar custom nodes in ComfyUI can lead to import conflicts or unexpected behavior due to overlapping functionalities and potentially outdated dependencies.","severity":"breaking","affected_versions":"Versions 0.0.7+"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Update `controlnet-aux` to the latest version: `pip install -U controlnet-aux`. If the issue persists, try reinstalling it in a fresh virtual environment or checking for conflicting `diffusers` versions (`pip list | grep diffusers`).","cause":"This error typically occurs when `controlnet-aux` is either outdated, or there's a conflict with other installed packages (e.g., an older version of `diffusers` or related utilities) that expect a different internal structure or API.","error":"ImportError: cannot import name 'resize_image_with_pad' from 'controlnet_aux.util'"},{"fix":"Check the console log immediately after starting ComfyUI for specific error messages regarding missing packages. Manually install any reported missing dependencies (`pip install <package_name>`). Ensure all requirements from `requirements.txt` (if applicable for a custom node wrapper) are met.","cause":"When `controlnet-aux` is integrated as a custom node in environments like ComfyUI, this often indicates missing Python dependencies required by specific annotators or a general installation issue preventing the nodes from loading.","error":"Nodes Not Appearing (in ComfyUI after installing controlnet_aux)"},{"fix":"Explicitly convert NumPy integers to standard Python integers using `int()` before passing them as arguments to functions that expect Python native types. For example, `(int(width), int(height))`.","cause":"This type error usually happens in image processing steps where a function expecting an integer or tuple of integers for dimensions receives NumPy integers (`numpy.int64`) instead, which are not implicitly cast.","error":"TypeError: expected size to be one of int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int], but got size with types [<class 'numpy.int64'>, <class 'numpy.int64'>]"}]}