Asynchronous File IO (caio)

raw JSON →
0.9.25 verified Tue May 12 auth: no python install: verified

caio is a Python library providing asynchronous file I/O for Linux, macOS, and Windows. It offers Python bindings for Linux AIO API, including `io_uring`, and provides fallback mechanisms for other platforms using threads or pure Python. Currently at version 0.9.25, the library is actively maintained with regular updates.

pip install caio
error ImportError: Error on io_setup with code 22
cause This error occurs when the `linux_aio` backend attempts to initialize the AIO context via `io_setup` but fails, often due to missing `libaio` development libraries or an incompatible kernel/filesystem.
fix
Install the libaio-dev (Debian/Ubuntu) or libaio (Red Hat/Fedora) package: sudo apt-get install libaio-dev or sudo yum install libaio. Ensure your Linux kernel is compatible (typically 4.18+ for native AIO) and the filesystem supports it. Alternatively, explicitly use a different backend like thread_aio or python_aio by setting the CAIO_IMPL environment variable (e.g., CAIO_IMPL=thread python your_app.py) or importing directly (e.g., from caio.thread_aio_asyncio import AsyncioContext).
error ImportError: io_uring_setup(2) returns ENOSYS (Operation not permitted)
cause This `ImportError` indicates that the `io_uring` system call is blocked, most commonly in containerized environments (Docker, Podman, Kubernetes) due to restrictive `seccomp` filters.
fix
Run your container with elevated security privileges that allow io_uring syscalls. For Docker/Podman, use docker run --security-opt seccomp=unconfined ... or provide a custom seccomp profile that permits io_uring_enter, io_uring_register, and io_uring_setup syscalls. For Kubernetes, configure securityContext.seccompProfile.type: Unconfined.
error ModuleNotFoundError: No module named 'caio'
cause This common Python error means the `caio` package is not installed in the Python environment being used, or the Python interpreter cannot find it in its search path.
fix
Ensure caio is installed in your active Python environment using pip: pip install caio. If using virtual environments, activate the correct environment before installation. Verify the Python interpreter being run is the one where caio was installed.
gotcha The `io_uring` backend might be blocked by `seccomp` filters in container environments like Docker, Podman, or Kubernetes. This can lead to `ImportError` when `io_uring_setup(2)` returns `ENOSYS`.
fix To fix this, run containers with `--security-opt seccomp=unconfined` (Docker/Podman) or set `securityContext.seccompProfile.type: Unconfined` (Kubernetes).
gotcha Native Linux AIO implementation requires a kernel version of 4.18 or newer. If an older kernel is detected, `caio` will fall back to a thread-based or pure Python implementation, which might have different performance characteristics.
fix Ensure your Linux kernel is 4.18 or newer for optimal performance with native AIO. Alternatively, explicitly select a backend using the `CAIO_IMPL` environment variable (`CAIO_IMPL=thread` or `CAIO_IMPL=python`) or by importing a specific backend directly (e.g., `from caio.thread_aio_asyncio import AsyncioContext`).
deprecated Direct imports of specific backend implementations (e.g., `from caio.linux_aio_asyncio import AsyncioContext`) were previously the primary way to force a backend. While still possible, it is now recommended to let `caio` pick the best available backend automatically via `from caio import AsyncioContext` or to use the `CAIO_IMPL` environment variable or a `default_implementation` file for global control.
fix Use `from caio import AsyncioContext` for automatic backend selection. For explicit control, set the `CAIO_IMPL` environment variable (e.g., `CAIO_IMPL=uring`) or create a `default_implementation` file with the desired backend name (e.g., `uring`).
python os / libc status wheel install import disk mem side effects
3.10 alpine (musl) wheel - 0.16s 17.9M 4.8M clean
3.10 alpine (musl) - - 0.20s 17.9M 4.8M -
3.10 slim (glibc) wheel 1.5s 0.12s 19M 4.8M clean
3.10 slim (glibc) - - 0.14s 19M 4.8M -
3.11 alpine (musl) wheel - 0.24s 19.7M 6.1M clean
3.11 alpine (musl) - - 0.32s 19.7M 6.1M -
3.11 slim (glibc) wheel 1.6s 0.22s 20M 6.1M clean
3.11 slim (glibc) - - 0.24s 20M 6.1M -
3.12 alpine (musl) wheel - 0.52s 11.6M 9.2M clean
3.12 alpine (musl) - - 0.57s 11.6M 9.2M -
3.12 slim (glibc) wheel 1.4s 0.47s 12M 9.2M clean
3.12 slim (glibc) - - 0.51s 12M 9.2M -
3.13 alpine (musl) wheel - 0.49s 11.4M 9.7M clean
3.13 alpine (musl) - - 0.60s 11.3M 9.7M -
3.13 slim (glibc) wheel 1.5s 0.45s 12M 9.7M clean
3.13 slim (glibc) - - 0.50s 12M 9.7M -
3.9 alpine (musl) build_error - - - - - -
3.9 alpine (musl) - - - - - -
3.9 slim (glibc) wheel 1.8s 0.15s 18M 4.6M clean
3.9 slim (glibc) - - 0.17s 18M 4.6M -

This quickstart demonstrates how to use `caio` with `asyncio` to perform basic asynchronous file operations like writing, reading, and synchronizing. It also shows how to execute multiple write operations concurrently. A temporary file is created and cleaned up for the demonstration.

import asyncio
import os
from caio import AsyncioContext

async def main():
    # Ensure a dummy file exists for the example
    file_path = "test.file"
    with open(file_path, "wb+") as f: # Create or truncate the file
        f.write(b"")

    ctx = AsyncioContext(max_requests=128)
    fd = os.open(file_path, os.O_RDWR | os.O_CREAT)

    try:
        # Execute one write operation
        await ctx.write(b"Hello world", fd, offset=0)
        print(f"Wrote: Hello world")

        # Execute one read operation
        read_data = await ctx.read(32, fd, offset=0)
        print(f"Read: {read_data.decode()}")

        # Execute one fdsync operation
        await ctx.fdsync(fd)
        print("File synchronized.")

        # Execute multiple writes concurrently
        op1 = ctx.write(b"Hello from ", fd, offset=0)
        op2 = ctx.write(b"async world", fd, offset=11)
        await asyncio.gather(op1, op2)
        print("Concurrent writes completed.")

        read_data_concurrent = await ctx.read(32, fd, offset=0)
        print(f"Read after concurrent writes: {read_data_concurrent.decode()}")

    finally:
        os.close(fd)
        os.remove(file_path)

if __name__ == '__main__':
    asyncio.run(main())