{"library":"pypeln","title":"PyPeln: Concurrent Data Pipelines","description":"PyPeln (pronounced \"pypeline\") is a Python library designed for building concurrent data pipelines with ease. It offers a simple, functional API that supports multiprocessing (processes), multithreading (threads), and asynchronous programming (asyncio tasks) with the same interface. This allows developers to create multi-stage pipelines and maintain fine-grained control over computational resources. The library is currently at version 0.4.9 and provides solutions for medium-scale data tasks where frameworks like Spark or Dask might be considered overkill.","language":"python","status":"active","last_verified":"Fri May 15","install":{"commands":["pip install pypeln"],"cli":null},"imports":["import pypeln as pl","from pypeln import process as pl_process","from pypeln import thread as pl_thread","from pypeln import task as pl_task"],"auth":{"required":false,"env_vars":[]},"quickstart":{"code":"import pypeln as pl\nimport time\nimport random\n\ndef slow_square(x):\n    time.sleep(random.uniform(0.1, 0.5)) # Simulate work\n    return x * x\n\ndef is_even(x):\n    return x % 2 == 0\n\ndef print_item(x):\n    print(f\"Processing: {x}\")\n    return x\n\ndata = range(10) # [0, 1, ..., 9]\n\n# Build a multiprocessing pipeline\n# 1. Map: square each number (4 workers)\n# 2. Filter: keep only even numbers (2 workers)\n# 3. Map: print the item (1 worker, ordered output)\nresults = (pl.process.map(slow_square, data, workers=4, maxsize=4) \n           .filter(is_even, workers=2, maxsize=2) \n           .map(print_item, workers=1, maxsize=1, ordered=True))\n\n# Consume the results (this starts the pipeline execution)\nfinal_list = list(results)\nprint(f\"Final result: {final_list}\")","lang":"python","description":"This quickstart demonstrates a basic multiprocessing pipeline. It defines a source iterable (`data`), then chains `map` and `filter` operations using `pl.process`. The pipeline squares numbers, filters for even ones, and prints each processed item. The `workers` and `maxsize` parameters control concurrency and backpressure for each stage. The `list()` call at the end triggers the execution and collects results.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":{"tag":null,"tag_description":null,"last_tested":"2026-05-15","installed_version":"0.4.9","pypi_latest":"0.4.9","is_stale":false,"summary":{"python_range":"3.10–3.9","success_rate":100,"avg_install_s":2.7,"avg_import_s":0.59,"wheel_type":"sdist"},"results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"noisy","install_time_s":null,"import_time_s":0.58,"mem_mb":11.2,"disk_size":"20.6M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"noisy","install_time_s":2.4,"import_time_s":0.4,"mem_mb":11.2,"disk_size":"21M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"noisy","install_time_s":null,"import_time_s":0.85,"mem_mb":12.4,"disk_size":"23.2M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"noisy","install_time_s":2.2,"import_time_s":0.7,"mem_mb":12.4,"disk_size":"24M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"broken","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":"12.9M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"broken","install_time_s":3.2,"import_time_s":null,"mem_mb":null,"disk_size":"13M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"broken","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":"12.6M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"broken","install_time_s":3,"import_time_s":null,"mem_mb":null,"disk_size":"13M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"noisy","install_time_s":null,"import_time_s":0.53,"mem_mb":10.9,"disk_size":"20.2M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"pypeln","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"noisy","install_time_s":2.5,"import_time_s":0.46,"mem_mb":10.9,"disk_size":"21M"}]}}