multitasking: Non-blocking Python methods using decorators

0.0.12 · active · verified Thu Apr 09

MultiTasking is a lightweight Python library, currently at version 0.0.12, designed to convert Python methods into asynchronous, non-blocking methods using simple decorators. It is particularly effective for I/O-bound tasks such as API calls, web scraping, and database queries, enabling concurrent operations without complex manual thread or process management. The library focuses on ease of use and aims for a stable, albeit infrequent, release cadence with a focus on improvements rather than breaking changes.

Common errors

```
ModuleNotFoundError: No module named 'multitasking'
```
cause The 'multitasking' library has not been installed in the Python environment.

fix Install the library using pip: `pip install multitasking`
```
AttributeError: 'module' object has no attribute 'task'
```
cause This error typically occurs if the 'task' decorator is misspelled or if a user attempts to call `multitasking.task()` as a function instead of using it as a decorator (`@multitasking.task`).

fix Ensure the decorator is spelled correctly and used as `@multitasking.task` above the function definition. For example: ```python import multitasking import time @multitasking.task def my_task(): time.sleep(1) print('Task finished') my_task() multitasking.wait_for_tasks() ```
```
Tasks not running concurrently
```
cause This is a common behavioral issue where decorated tasks appear to run sequentially rather than in parallel, often due to `multitasking.wait_for_tasks()` being called prematurely within a loop or immediately after each task initiation, preventing true concurrency.

fix Ensure that `multitasking.wait_for_tasks()` is called only after all desired tasks have been initiated, allowing them to run concurrently in the background. For example: ```python import multitasking import time @multitasking.task def my_task(task_id): print(f'Starting task {task_id}') time.sleep(2) print(f'Finished task {task_id}') for i in range(3): my_task(i) # Initiate tasks without waiting print('All tasks initiated, waiting for completion...') multitasking.wait_for_tasks() # Wait for all initiated tasks to finish print('All tasks completed') ```

Warnings

gotcha Python's Global Interpreter Lock (GIL) limits true parallelism for CPU-bound tasks. While `multitasking` uses threads by default to achieve concurrency, CPU-intensive tasks will not run in parallel across multiple CPU cores due to the GIL, potentially leading to performance degradation from context switching rather than gains.
Fix: For CPU-bound tasks, consider configuring `multitasking` to use `multiprocessing.Process` instead of `threading.Thread` by using `multitasking.set_engine('process')`. Be aware that multiprocessing introduces higher overhead for startup and inter-process communication.
gotcha By default, `multitasking` uses Python's `threading` module, which is efficient for I/O-bound operations (e.g., network requests, file I/O) where threads spend most of their time waiting for external resources. However, it's not optimal for CPU-bound tasks due to the GIL.
Fix: Ensure that `multitasking` is primarily used for I/O-bound workloads to maximize its benefits. For CPU-bound tasks requiring true parallelism, explicitly switch the execution engine to multiprocessing using `multitasking.set_engine('process')` before launching tasks, if the overhead is acceptable for your application.
gotcha The default maximum number of threads is typically based on the number of CPU cores. While `multitasking` automatically manages a pool, this default might not be optimal for all scenarios, especially when dealing with a very high number of short-lived I/O-bound tasks.
Fix: Adjust the maximum number of concurrent tasks using `multitasking.set_max_threads(N)` to match your application's specific needs and resource availability. Experiment with different values to find the sweet spot for your workload.

Install

pip install multitasking Install latest version

Imports

multitasking
```
import multitasking
```
Commonly imported directly or specific components are imported from it.
task
```
from multitasking import task
```
Decorator for making a function non-blocking.
wait_for_tasks
```
from multitasking import wait_for_tasks
```
Function to block until all `multitasking.task` decorated functions complete.

Quickstart

This quickstart demonstrates how to use the `@multitasking.task` decorator to make a function non-blocking. The `fetch_data` calls will run concurrently. `multitasking.wait_for_tasks()` is used to pause the main thread until all decorated tasks have finished executing. This pattern is ideal for I/O-bound operations where the main program shouldn't wait for each individual task to complete.

import multitasking
import time

@multitasking.task
def fetch_data(url_id):
    # Simulate API call or I/O operation
    time.sleep(1)
    print(f"Fetched data from URL {url_id}")
    return f"Data from {url_id}"

if __name__ == "__main__":
    print("Starting tasks...")
    for i in range(5):
        fetch_data(i)

    # Wait for all tasks to complete
    multitasking.wait_for_tasks()
    print("All tasks completed!")

view raw JSON →