Queuelib

1.9.0 · active · verified Fri Apr 10

Queuelib is a Python library providing a collection of persistent (disk-based) and non-persistent (memory-based) queue implementations. It's often used in projects requiring efficient data storage and retrieval, particularly in contexts like web crawling (e.g., Scrapy). The current version is 1.9.0, and it maintains an active, though irregular, release cadence, primarily focusing on Python version compatibility and minor enhancements.

Warnings

breaking Queuelib frequently removes support for older Python versions. Ensure your environment meets the `requires_python` specification for the installed version.
Fix: Upgrade your Python interpreter to a supported version (e.g., Python 3.10+ for queuelib 1.9.0) or downgrade queuelib if older Python support is critical.
gotcha Disk-based queues (e.g., `FifoDiskQueue`, `LifoDiskQueue`, `PriorityDiskQueue`) *must* be explicitly closed using the `.close()` method to ensure all pending writes are flushed to disk and file handles are released. Failure to close can lead to data loss or corruption, especially during unexpected program termination.
Fix: Always call `.close()` on disk queue instances, preferably within a `try...finally` block or by using them as context managers if available (though explicit `.close()` is the most robust approach for Queuelib).
gotcha The `peek()` method, which allows viewing the next item without removing it, was added in version 1.6.0. Code relying on `peek()` will fail on older `queuelib` versions.
Fix: Upgrade `queuelib` to version 1.6.0 or newer if you need `peek()` functionality. If you cannot upgrade, you'll need to implement alternative logic.
gotcha Be mindful of the distinct import paths for different queue types. `PriorityQueue` (in-memory) is imported directly from `queuelib`, while persistent disk queues are in `queuelib.disk`, and other memory queues are in `queuelib.queue`. Confusing these paths is a common mistake.
Fix: Always refer to the official documentation or source code for the correct import path for the specific queue class you intend to use (e.g., `from queuelib.disk import FifoDiskQueue` vs. `from queuelib import PriorityQueue`).

Install

pip install queuelib Install stable version

Imports

FifoDiskQueue
```
from queuelib.disk import FifoDiskQueue
```
Disk-based queues are in the 'queuelib.disk' submodule, not directly from the top-level package.
PriorityQueue
```
from queuelib import PriorityQueue
```
The default in-memory PriorityQueue is directly available from the top-level 'queuelib' package. Other memory queues (FifoMemoryQueue, LifoMemoryQueue) are in 'queuelib.queue'.

Quickstart

This quickstart demonstrates creating and using a `FifoDiskQueue` for persistent storage. It shows pushing and popping items, closing the queue for integrity, and reopening it to retrieve previously stored data. Remember to always call `.close()` on disk-based queues and handle the storage directory appropriately.

import tempfile
import shutil
from queuelib.disk import FifoDiskQueue

# Create a temporary directory for the queue files
qdir = tempfile.mkdtemp()
try:
    q = FifoDiskQueue(qdir)
    print(f"Queue created at: {qdir}")

    q.push(b'item1')
    q.push(b'item2')
    print(f"Pushed: item1, item2. Size: {len(q)}")

    item = q.pop()
    print(f"Popped: {item}. Size: {len(q)}")

    q.close() # IMPORTANT: Always close disk queues to ensure data integrity
    print("Queue closed.")

    # Reopen the queue from the same directory to demonstrate persistence
    q2 = FifoDiskQueue(qdir)
    item2 = q2.pop()
    print(f"Reopened queue. Popped: {item2}. Size: {len(q2)}")
    q2.close()

except Exception as e:
    print(f"An error occurred: {e}")
finally:
    shutil.rmtree(qdir) # Clean up the temporary directory
    print(f"Cleaned up directory: {qdir}")

view raw JSON →