multitasking: Non-blocking Python methods using decorators

0.0.12 · active · verified Thu Apr 09

MultiTasking is a lightweight Python library, currently at version 0.0.12, designed to convert Python methods into asynchronous, non-blocking methods using simple decorators. It is particularly effective for I/O-bound tasks such as API calls, web scraping, and database queries, enabling concurrent operations without complex manual thread or process management. The library focuses on ease of use and aims for a stable, albeit infrequent, release cadence with a focus on improvements rather than breaking changes.

Warnings

gotcha Python's Global Interpreter Lock (GIL) limits true parallelism for CPU-bound tasks. While `multitasking` uses threads by default to achieve concurrency, CPU-intensive tasks will not run in parallel across multiple CPU cores due to the GIL, potentially leading to performance degradation from context switching rather than gains.
Fix: For CPU-bound tasks, consider configuring `multitasking` to use `multiprocessing.Process` instead of `threading.Thread` by using `multitasking.set_engine('process')`. Be aware that multiprocessing introduces higher overhead for startup and inter-process communication.
gotcha By default, `multitasking` uses Python's `threading` module, which is efficient for I/O-bound operations (e.g., network requests, file I/O) where threads spend most of their time waiting for external resources. However, it's not optimal for CPU-bound tasks due to the GIL.
Fix: Ensure that `multitasking` is primarily used for I/O-bound workloads to maximize its benefits. For CPU-bound tasks requiring true parallelism, explicitly switch the execution engine to multiprocessing using `multitasking.set_engine('process')` before launching tasks, if the overhead is acceptable for your application.
gotcha The default maximum number of threads is typically based on the number of CPU cores. While `multitasking` automatically manages a pool, this default might not be optimal for all scenarios, especially when dealing with a very high number of short-lived I/O-bound tasks.
Fix: Adjust the maximum number of concurrent tasks using `multitasking.set_max_threads(N)` to match your application's specific needs and resource availability. Experiment with different values to find the sweet spot for your workload.

Install

pip install multitasking Install latest version

Imports

multitasking
```
import multitasking
```
Commonly imported directly or specific components are imported from it.
task
```
from multitasking import task
```
Decorator for making a function non-blocking.
wait_for_tasks
```
from multitasking import wait_for_tasks
```
Function to block until all `multitasking.task` decorated functions complete.

Quickstart

This quickstart demonstrates how to use the `@multitasking.task` decorator to make a function non-blocking. The `fetch_data` calls will run concurrently. `multitasking.wait_for_tasks()` is used to pause the main thread until all decorated tasks have finished executing. This pattern is ideal for I/O-bound operations where the main program shouldn't wait for each individual task to complete.

import multitasking
import time

@multitasking.task
def fetch_data(url_id):
    # Simulate API call or I/O operation
    time.sleep(1)
    print(f"Fetched data from URL {url_id}")
    return f"Data from {url_id}"

if __name__ == "__main__":
    print("Starting tasks...")
    for i in range(5):
        fetch_data(i)

    # Wait for all tasks to complete
    multitasking.wait_for_tasks()
    print("All tasks completed!")

view raw JSON →