{"id":8943,"library":"django-chunkator","title":"django-chunkator","description":"django-chunkator is a Python library for Django that allows iterating over large QuerySets in small, memory-efficient chunks, preventing high RAM consumption. The current version is 2.0.0, and the project actively maintains compatibility with recent Django and Python versions.","status":"active","version":"2.0.0","language":"en","source_language":"en","source_url":"https://github.com/peopledoc/django-chunkator","tags":["django","queryset","memory-management","pagination","iteration","database"],"install":[{"cmd":"pip install django-chunkator","lang":"bash","label":"Install stable version"}],"dependencies":[],"imports":[{"symbol":"chunkator","correct":"from chunkator import chunkator"},{"note":"Use `chunkator_page` to get pages of items directly, useful for background tasks.","symbol":"chunkator_page","correct":"from chunkator import chunkator_page"}],"quickstart":{"code":"import os\nimport django\nfrom django.conf import settings\nfrom django.db import models\n\n# Minimal Django setup for demonstration\nif not settings.configured:\n    settings.configure(\n        DEBUG=True,\n        INSTALLED_APPS=[\n            'django.contrib.auth',\n            'django.contrib.contenttypes',\n            'your_app_name', # Placeholder\n        ],\n        DATABASES={'default': {'ENGINE': 'django.db.backends.sqlite3', 'NAME': ':memory:'}},\n    )\ndjango.setup()\n\nclass LargeModel(models.Model):\n    name = models.CharField(max_length=255)\n    value = models.IntegerField()\n\n    class Meta:\n        app_label = 'your_app_name'\n\n    def __str__(self):\n        return self.name\n\n# Create a mock app label if needed for standalone execution\ntry:\n    from django.apps import apps\n    apps.get_app_config('your_app_name')\nexcept LookupError:\n    class YourAppConfig(django.apps.AppConfig):\n        name = 'your_app_name'\n        label = 'your_app_name'\n    apps.apps_ready.connect(YourAppConfig.ready)\n\n# Example usage\nfrom chunkator import chunkator\n\n# To make the model usable, typically in a real Django project,\n# you would have migrations and a database.\n# For this quickstart, we'll bypass real DB creation and just demonstrate iteration.\n# In a real scenario, you'd have LargeModel.objects.all()\n\n# Simulate a QuerySet\nclass MockQuerySet:\n    def __init__(self, data):\n        self._data = data\n    \n    def all(self):\n        return self\n\n    def __iter__(self):\n        yield from self._data\n    \n    def order_by(self, *args, **kwargs):\n        return self # For demonstration, ignore ordering\n\n    def values(self, *fields):\n        if 'pk' not in fields and 'id' not in fields:\n            raise ValueError(\"Mock MissingPkFieldException: 'pk' must be included in values() call.\")\n        return MockQuerySet([{\n            field: getattr(item, field) if hasattr(item, field) else item.id if field == 'pk' else None\n            for field in fields\n        } for item in self._data])\n\nmock_data = []\nfor i in range(1, 101):\n    obj = LargeModel(id=i, name=f'Item {i}', value=i*10)\n    obj.pk = i # Ensure pk is set for mock\n    mock_data.append(obj)\n\nmock_queryset = MockQuerySet(mock_data)\n\nprint(\"Iterating with chunkator (objects):\")\nfor item in chunkator(mock_queryset, 20):\n    # In a real Django app, item would be a model instance\n    # print(f\"Processing item: {item.name}\")\n    pass # do something with item\nprint(\"Done iterating objects.\")\n\nprint(\"\\nIterating with chunkator (values with pk):\")\nfor item_dict in chunkator(mock_queryset.values('pk', 'name'), 25):\n    # item_dict is a dictionary if .values() is used\n    # print(f\"Processing item dict: {item_dict['name']}\")\n    pass # do something with item_dict\nprint(\"Done iterating values.\")","lang":"python","description":"Demonstrates how to use `chunkator` to iterate over a large Django QuerySet in manageable chunks. Note that for `values()` queries, the primary key field (e.g., 'pk' or 'id') must be explicitly included to avoid `MissingPkFieldException`."},"warnings":[{"fix":"Ensure your project is running on Python 3.5 or newer. Upgrade your Python environment.","message":"Python 2 support has been officially dropped as of version 2.0.0.","severity":"breaking","affected_versions":">=2.0.0"},{"fix":"Upgrade your Django project to a supported version (e.g., Django 2.2, 3.0 or higher compatible with Python 3.5+).","message":"Support for older Django versions (1.6, 1.7, 1.11) has been progressively removed. Version 2.0.0 specifically drops Django 1.11 support.","severity":"breaking","affected_versions":">=2.0.0"},{"fix":"Modify your `values()` call to include the primary key, e.g., `LargeModel.objects.values('pk', 'name')`.","message":"When using `queryset.values()` with `chunkator`, you MUST include the primary key field (e.g., 'pk' or 'id') in the `values()` call, otherwise a `MissingPkFieldException` will be raised.","severity":"gotcha","affected_versions":"All"},{"fix":"Understand that the primary benefit is memory efficiency. If performance is the sole concern, analyze your database queries and indexes, or consider Django's built-in `.iterator()` for server-side cursors (if supported and enabled by your DB backend).","message":"`django-chunkator` is designed to reduce RAM usage, not to accelerate query processing. It converts one large query into several smaller queries, which can sometimes be slower overall but prevents out-of-memory errors for huge datasets.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Include the primary key field in your `.values()` call: `from chunkator import chunkator; for item in chunkator(MyModel.objects.values('id', 'other_field'), 100): ...`","cause":"Attempting to use `chunkator` with a `QuerySet.values()` call that does not include the primary key field (e.g., 'pk' or 'id').","error":"MissingPkFieldException"},{"fix":"Upgrade your Python environment to Python 3.5 or a newer compatible version. If you must use Python 2, you need to downgrade `django-chunkator` to a pre-2.0.0 version (e.g., 1.5.0).","cause":"Running `django-chunkator` version 2.0.0 or higher on a Python 2 environment.","error":"Python 2 is no longer supported by django-chunkator 2.0.0."}]}