{"id":2744,"library":"queuelib","title":"Queuelib","description":"Queuelib is a Python library providing a collection of persistent (disk-based) and non-persistent (memory-based) queue implementations. It's often used in projects requiring efficient data storage and retrieval, particularly in contexts like web crawling (e.g., Scrapy). The current version is 1.9.0, and it maintains an active, though irregular, release cadence, primarily focusing on Python version compatibility and minor enhancements.","status":"active","version":"1.9.0","language":"en","source_language":"en","source_url":"https://github.com/scrapy/queuelib","tags":["queue","persistent","memory","scrapy","data structures","fifo","lifo","priority"],"install":[{"cmd":"pip install queuelib","lang":"bash","label":"Install stable version"}],"dependencies":[],"imports":[{"note":"Disk-based queues are in the 'queuelib.disk' submodule, not directly from the top-level package.","wrong":"from queuelib import FifoDiskQueue","symbol":"FifoDiskQueue","correct":"from queuelib.disk import FifoDiskQueue"},{"note":"The default in-memory PriorityQueue is directly available from the top-level 'queuelib' package. Other memory queues (FifoMemoryQueue, LifoMemoryQueue) are in 'queuelib.queue'.","wrong":"from queuelib.queue import PriorityQueue","symbol":"PriorityQueue","correct":"from queuelib import PriorityQueue"}],"quickstart":{"code":"import tempfile\nimport shutil\nfrom queuelib.disk import FifoDiskQueue\n\n# Create a temporary directory for the queue files\nqdir = tempfile.mkdtemp()\ntry:\n    q = FifoDiskQueue(qdir)\n    print(f\"Queue created at: {qdir}\")\n\n    q.push(b'item1')\n    q.push(b'item2')\n    print(f\"Pushed: item1, item2. Size: {len(q)}\")\n\n    item = q.pop()\n    print(f\"Popped: {item}. Size: {len(q)}\")\n\n    q.close() # IMPORTANT: Always close disk queues to ensure data integrity\n    print(\"Queue closed.\")\n\n    # Reopen the queue from the same directory to demonstrate persistence\n    q2 = FifoDiskQueue(qdir)\n    item2 = q2.pop()\n    print(f\"Reopened queue. Popped: {item2}. Size: {len(q2)}\")\n    q2.close()\n\nexcept Exception as e:\n    print(f\"An error occurred: {e}\")\nfinally:\n    shutil.rmtree(qdir) # Clean up the temporary directory\n    print(f\"Cleaned up directory: {qdir}\")","lang":"python","description":"This quickstart demonstrates creating and using a `FifoDiskQueue` for persistent storage. It shows pushing and popping items, closing the queue for integrity, and reopening it to retrieve previously stored data. Remember to always call `.close()` on disk-based queues and handle the storage directory appropriately."},"warnings":[{"fix":"Upgrade your Python interpreter to a supported version (e.g., Python 3.10+ for queuelib 1.9.0) or downgrade queuelib if older Python support is critical.","message":"Queuelib frequently removes support for older Python versions. Ensure your environment meets the `requires_python` specification for the installed version.","severity":"breaking","affected_versions":">=1.7.0"},{"fix":"Always call `.close()` on disk queue instances, preferably within a `try...finally` block or by using them as context managers if available (though explicit `.close()` is the most robust approach for Queuelib).","message":"Disk-based queues (e.g., `FifoDiskQueue`, `LifoDiskQueue`, `PriorityDiskQueue`) *must* be explicitly closed using the `.close()` method to ensure all pending writes are flushed to disk and file handles are released. Failure to close can lead to data loss or corruption, especially during unexpected program termination.","severity":"gotcha","affected_versions":"All versions with disk queues"},{"fix":"Upgrade `queuelib` to version 1.6.0 or newer if you need `peek()` functionality. If you cannot upgrade, you'll need to implement alternative logic.","message":"The `peek()` method, which allows viewing the next item without removing it, was added in version 1.6.0. Code relying on `peek()` will fail on older `queuelib` versions.","severity":"gotcha","affected_versions":"<1.6.0"},{"fix":"Always refer to the official documentation or source code for the correct import path for the specific queue class you intend to use (e.g., `from queuelib.disk import FifoDiskQueue` vs. `from queuelib import PriorityQueue`).","message":"Be mindful of the distinct import paths for different queue types. `PriorityQueue` (in-memory) is imported directly from `queuelib`, while persistent disk queues are in `queuelib.disk`, and other memory queues are in `queuelib.queue`. Confusing these paths is a common mistake.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-10T00:00:00.000Z","next_check":"2026-07-09T00:00:00.000Z"}