Qwen Omni Language Model Utilities

Common errors

• KeyError: 'qwen2_5_omni'

  cause This error typically occurs when the `transformers` library does not have the necessary Qwen Omni model configurations, often due to an outdated or incompatible version of `transformers` with the specific Qwen Omni model being loaded.
  fix Ensure you are using the `transformers` version explicitly recommended by the Qwen model's documentation or Hugging Face page. This might involve uninstalling your current `transformers` and installing a specific version or from a particular Git branch, e.g., `pip uninstall transformers && pip install git+https://github.com/huggingface/transformers@v4.51.3-Qwen2.5-Omni-preview`.

• Video processing is slow or fails.

  cause Often due to `decord` not being installed or `ffmpeg` not being available on the system. If `decord` isn't installed, `qwen-omni-utils` falls back to `torchvision`, which can be less performant for video.
  fix First, ensure `ffmpeg` is installed on your system (`sudo apt-get install ffmpeg` on Debian/Ubuntu). Then, install `qwen-omni-utils` with the `decord` extra: `pip install qwen-omni-utils[decord] -U`. If `decord` fails to install on your OS (e.g., Windows/macOS), you may need to compile it from source or use the `torchvision` fallback with potentially reduced performance.

• ModuleNotFoundError: No module named 'qwen_omni_utils'

  cause The `qwen-omni-utils` package is not installed in your current Python environment.
  fix Install the package using pip: `pip install qwen-omni-utils` or `pip install qwen-omni-utils[decord]` for full functionality. If using a virtual environment, ensure it is activated before installation.

Warnings

• breaking The GitHub repository for `qwen-omni-utils` can be significantly out of sync with the PyPI release. New features or fixes present in the PyPI package might not be reflected in the public GitHub source code for the utility, leading to confusion when reviewing source or contributing.
  Fix: Always install the PyPI version (`pip install qwen-omni-utils`) for the most up-to-date and functional code. Refer to PyPI for declared dependencies and changelogs over GitHub source for `qwen-omni-utils` itself.
• gotcha Strict `transformers` library version compatibility is often required for Qwen Omni models. Using an incompatible `transformers` version can lead to `KeyError: 'qwen2_5_omni'` or other model loading failures.
  Fix: Refer to the specific Qwen model's Hugging Face page or documentation for the exact recommended `transformers` version. Often, installing `transformers` from a specific GitHub branch or commit is advised, e.g., `pip install git+https://github.com/huggingface/transformers@v4.51.3-Qwen2.5-Omni-preview`.
• gotcha `decord` for faster video loading might not install correctly from PyPI on non-Linux systems. If `decord` installation fails, `qwen-omni-utils` will fall back to `torchvision` for video processing, which might be slower.
  Fix: On non-Linux systems, either accept `torchvision` as a fallback or compile `decord` from source if optimal video performance is critical. Ensure `ffmpeg` is installed, as it's a general prerequisite for multimedia handling.
• gotcha When integrating `qwen-omni-utils` with `vLLM` for inference, users have reported issues where text generation either cuts off abruptly or enters an infinite repetition loop. This is linked to internal differences in how `positions`, `eager`, and `CUDA` parameters are handled within `Qwen2Attention` module in `vLLM`.
  Fix: Monitor `vLLM` and Qwen model GitHub issues for updates and specific patches related to `Qwen2Attention` module compatibility. If encountering this, try different `vLLM` versions or avoid streaming generation with `qwen-omni-utils` if possible.

Install

• pip install qwen-omni-utils -U Basic Installation
• pip install qwen-omni-utils[decord] -U sudo apt-get install ffmpeg Recommended for Faster Video (Linux)

Imports

• process_mm_info

  from qwen_omni_utils import process_mm_info

  The primary utility function for preparing multimodal inputs for Qwen models.

Quickstart

This quickstart demonstrates how to import `process_mm_info` from `qwen_omni_utils`. While it shows how model and processor loading would typically be done, the actual heavy model loading and inference steps are commented out due to resource requirements. The `process_mm_info` function is key for preparing diverse multimodal inputs for the Qwen Omni models. Ensure you have `ffmpeg` installed for full video capabilities and a compatible `transformers` version.

import soundfile as sf
import torch
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info
import os

# NOTE: Replace with your actual model path or Hugging Face model ID
model_id = "Qwen/Qwen2.5-Omni-7B"

# Ensure you have a Hugging Face token if using private models
# os.environ['HF_TOKEN'] = os.environ.get('HF_TOKEN', 'hf_YOUR_TOKEN_HERE') 

# Load model and processor (requires significant GPU memory)
# model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
#     model_id, torch_dtype="auto", device_map="auto"
# )
# processor = Qwen2_5OmniProcessor.from_pretrained(model_id)

# Example usage with process_mm_info (assuming model/processor loaded above)
# This function prepares multimodal content for the processor.
# content = [
#     {"type": "text", "text": "Describe this image:"},
#     {"type": "image", "image": "https://example.com/image.jpg"},
#     {"type": "text", "text": "And tell me about this audio:"},
#     {"type": "audio", "audio": "https://example.com/audio.wav"}
# ]
# processed_content = process_mm_info(content, processor)

print("qwen-omni-utils is successfully imported and ready to process multimodal inputs.")
print("Refer to Qwen model documentation for full model loading and inference examples.")