{"id":3333,"library":"luigi","title":"Luigi Workflow Management","description":"Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, and much more. It's developed by Spotify and is currently in version 3.8.0, with minor releases typically occurring every few months.","status":"active","version":"3.8.0","language":"en","source_language":"en","source_url":"https://github.com/spotify/luigi","tags":["workflow-orchestration","ETL","task-scheduling","data-pipeline","batch-processing"],"install":[{"cmd":"pip install luigi","lang":"bash","label":"Install Luigi"}],"dependencies":[],"imports":[{"symbol":"Task","correct":"import luigi\n\nclass MyTask(luigi.Task): ..."},{"symbol":"Parameter","correct":"import luigi\n\nclass MyTask(luigi.Task):\n    param = luigi.Parameter()"},{"symbol":"LocalTarget","correct":"import luigi\n\nluigi.LocalTarget('path/to/file')"},{"symbol":"build","correct":"import luigi\n\nluigi.build([my_task_instance], local_scheduler=True)"}],"quickstart":{"code":"import luigi\nimport datetime\nimport os\n\nclass GenerateReport(luigi.Task):\n    date = luigi.DateParameter(default=datetime.date.today())\n\n    def run(self):\n        # Simulate some data processing\n        report_content = f\"Daily Report for {self.date.isoformat()}\\n\" \\\n                         f\"Generated by Luigi.\\n\"\n\n        # Write the report to a target file\n        with self.output().open('w') as f:\n            f.write(report_content)\n\n    def output(self):\n        # Define where the output of this task will be stored\n        # Using an environment variable for flexibility or defaulting to current dir\n        output_dir = os.environ.get('LUIGI_OUTPUT_DIR', '.')\n        return luigi.LocalTarget(os.path.join(output_dir, f'report_{self.date.isoformat()}.txt'))\n\nif __name__ == '__main__':\n    # To run this task using the command line (most common):\n    # 1. Start the Luigi scheduler daemon in a separate terminal: luigid --port 8082\n    # 2. Run your script: python your_script_name.py GenerateReport --date 2023-10-26\n    #    (or omit --date for today's date if default is set)\n    # 3. If you don't want to run luigid, use the --local-scheduler flag:\n    #    python your_script_name.py GenerateReport --local-scheduler\n\n    # For programmatic execution (e.g., in a wrapper script or test):\n    # The 'local_scheduler=True' ensures it runs without an external luigid daemon.\n    luigi.build([GenerateReport(date=datetime.date(2023, 10, 26))], local_scheduler=True)","lang":"python","description":"This quickstart defines a simple Luigi task `GenerateReport` that takes a `date` parameter, simulates generating a report, and writes it to a local file. It demonstrates defining a task, its `run` method, and its `output` target using `luigi.LocalTarget`. The example also shows how to trigger tasks programmatically using `luigi.build` with a local scheduler."},"warnings":[{"fix":"Upgrade your Python environment to a compatible version (3.10, 3.11, 3.12, 3.13) before upgrading Luigi to 3.x.","message":"Luigi 3.x series requires Python >= 3.10 and < 3.14. Projects on older Python 3 versions (e.g., 3.6-3.9) or newer versions (3.14+) will not be compatible.","severity":"breaking","affected_versions":"3.x.x"},{"fix":"Start `luigid` in a separate terminal (`luigid --port 8082`) or always include `--local-scheduler` in your command line execution or `local_scheduler=True` in `luigi.build()` calls.","message":"Luigi tasks require a scheduler to run. By default, it looks for a running `luigid` daemon. For simple local execution or development, remember to use the `--local-scheduler` flag when running tasks from the command line, or `local_scheduler=True` when using `luigi.build()` programmatically.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review custom plugins or extensions for any direct calls or implicit dependencies on `pkg_resources` and migrate to `importlib.metadata` (for Python 3.8+) or alternative methods.","message":"The `pkg_resources` library, which was historically used for introspection and entry points, has been removed in Luigi 3.7.3. While this is primarily an internal change, it might affect custom plugins or integrations that directly or indirectly relied on Luigi's usage of `pkg_resources`.","severity":"deprecated","affected_versions":">=3.7.3"},{"fix":"Ensure that every `run()` method reliably creates the target specified by its `output()` method. For tasks that don't produce physical outputs, consider using a dummy `LocalTarget('/tmp/luigi_flag_file')` or `luigi.MockTarget` for testing, or ensure their dependencies cover the actual work.","message":"Luigi determines task completion solely based on whether the `output()` targets `exist()`. If a task's `run()` method completes but fails to create its defined `output()`, Luigi will consider it incomplete and re-run it in subsequent invocations. Similarly, manually deleting an output file will cause Luigi to re-run the producing task.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-11T00:00:00.000Z","next_check":"2026-07-10T00:00:00.000Z"}