Image Deduplicator

0.3.3.post2 · active · verified Sat Apr 11

Imagededup is a Python package that simplifies finding exact and near-duplicate images in a collection. It offers various algorithms like perceptual hashing (PHash, DHash, WHash, AHash) and convolutional neural networks (CNNs) for robust deduplication. The package also includes an evaluation framework and plotting utilities for duplicates. The current version is 0.3.3.post2, and it maintains an active development cadence.

Warnings

breaking Python 3.8 and older versions are no longer supported. Imagededup now requires Python 3.9 or higher.
Fix: Upgrade your Python environment to 3.9 or newer.
breaking CNN encodings size has changed from 1024 to 576, and the underlying network might differ. This means CNN encodings from previous versions are incompatible.
Fix: Re-generate CNN encodings for your image collection when upgrading from older versions. Also, similarity thresholds might need adjustment for optimal performance with the new encodings.
breaking Hashes (all types) generated by versions >=0.3.0 may differ from previous versions for a given image, making older hash maps incompatible.
Fix: Re-generate all image hashes if upgrading from a version older than 0.3.0.
gotcha When using CNNs for deduplication, memory usage can increase quadratically with the number of images during retrieval (cosine similarity matrix calculation).
Fix: For very large datasets, consider using approximate nearest neighbor libraries (e.g., FAISS) or alternative strategies if memory becomes a bottleneck. Benchmarking with your specific dataset is recommended.
gotcha The optimal `min_similarity_threshold` for CNN and `max_distance_threshold` for hashing methods can vary significantly based on your dataset and the type of duplicates (exact vs. near-duplicates) you are trying to find.
Fix: Experiment with different threshold values for your specific dataset. The documentation suggests `min_similarity_threshold` around 0.9 for CNNs and `max_distance_threshold` of 0 for exact duplicates with hashing methods.

Install

pip install imagededup Install from PyPI

Imports

PHash
```
from imagededup.methods import PHash
```
DHash
```
from imagededup.methods import DHash
```
WHash
```
from imagededup.methods import WHash
```
AHash
```
from imagededup.methods import AHash
```
CNN
```
from imagededup.methods import CNN
```

plot_duplicates

from imagededup.utils import plot_duplicates

Quickstart

This quickstart demonstrates how to use the Perceptual Hashing (PHash) method to find duplicate images in a specified directory. It involves initializing the hashing method, generating image encodings, and then finding duplicates based on these encodings. The example includes creating a dummy directory and files for immediate execution.

import os
from imagededup.methods import PHash

# Create a dummy image directory and some dummy files for demonstration
# In a real scenario, 'image_dir' would point to your directory of images.
image_dir = './my_images'
os.makedirs(image_dir, exist_ok=True)
with open(os.path.join(image_dir, 'image1.jpg'), 'w') as f: f.write('dummy image content 1')
with open(os.path.join(image_dir, 'image2.png'), 'w') as f: f.write('dummy image content 2')
# Add more dummy images as needed to test deduplication

# Initialize a perceptual hashing method
phasher = PHash()

# Generate encodings for all images in the directory
encodings = phasher.encode_images(image_dir=image_dir)

# Find duplicates using the generated encodings
duplicates = phasher.find_duplicates(encoding_map=encodings)

print(f"Found duplicates: {duplicates}")

# Optionally, remove the dummy directory and files
# import shutil
# shutil.rmtree(image_dir)

view raw JSON →