IterableIO: Adapt Generators to File-like Objects

1.0.1 · active · verified Fri Apr 17

iterable-io is a lightweight Python library (current version 1.0.1) that provides an `IterableIO` class to adapt any byte-iterable (like generators, lists, or custom iterators) into a file-like object conforming to the `io.RawIOBase` interface. This allows iterables to be used in contexts expecting file-like objects, such as `zipfile.ZipFile`, `tarfile.TarFile`, or HTTP request bodies. The library is stable with a slow, focused release cadence.

Common errors

```
TypeError: a bytes-like object is required, not 'str'
```
cause Attempting to pass an iterable of Python `str` objects directly to `IterableIO` without encoding them to `bytes`.

fix Encode your string iterable to bytes: `IterableIO(s.encode('utf-8') for s in string_iterable)` or wrap with `io.TextIOWrapper` after creating the `IterableIO` from encoded bytes.
```
io.UnsupportedOperation: not writable
```
cause Calling a write method (e.g., `write()`, `writelines()`) on an `IterableIO` instance.

fix The `IterableIO` class is designed for reading from iterables. It does not support writing. If you need a writable file-like object, use `io.BytesIO` or `io.StringIO`.
```
TypeError: 'int' object is not iterable
```
cause Passing a non-iterable object (like an integer or a string literal) directly to `IterableIO` instead of an iterable (e.g., list, tuple, generator).

fix Ensure the argument passed to `IterableIO` is an object that can be iterated over, e.g., `IterableIO([b'chunk1', b'chunk2'])` or `IterableIO(my_generator())`.

Warnings

gotcha `IterableIO` expects an iterable of bytes. If you pass an iterable of strings, you must encode them yourself (e.g., `(s.encode('utf-8') for s in string_iterable)`) or wrap the `IterableIO` instance with `io.TextIOWrapper`.
Fix: Ensure the underlying iterable yields `bytes` objects, or use `io.TextIOWrapper(IterableIO(byte_iterable), encoding='utf-8')` for text handling.
gotcha `IterableIO` instances are read-only by default. Attempts to call `write()` or related methods will raise an `io.UnsupportedOperation`.
Fix: This library is designed for reading from iterables. If write functionality is needed, a different approach or library should be used, or a custom class inheriting from `io.RawIOBase` that implements writable methods.
gotcha The `read(size)` method may return less than `size` bytes if the underlying iterable yields a smaller chunk or is exhausted. It returns an empty bytes object `b''` only when the iterable is fully consumed.
Fix: Always check the return value of `read()` for `b''` to detect end-of-file, rather than assuming it will return exactly `size` bytes or `None`.

Install

pip install iterable-io Install latest version

Imports

IterableIO
```
from iterable_io import IterableIO
```

Quickstart

Demonstrates creating an `IterableIO` from a byte generator and using it directly or wrapping it with `io.TextIOWrapper` for string-based operations.

from iterable_io import IterableIO
import io

# Example 1: Adapting a generator of bytes
def byte_generator():
    yield b"Hello"
    yield b" "
    yield b"World!"

# Create an IterableIO object from the generator
file_like_object = IterableIO(byte_generator())

# Use it like a file
print(f"Read 6 bytes: {file_like_object.read(6).decode()}") # Output: Hello 
print(f"Read remaining: {file_like_object.read().decode()}") # Output: World!

# Example 2: Adapting string data using io.TextIOWrapper
string_data = ["line1\n", "line2\n", "last line"]

# The IterableIO expects bytes, so encode the strings
byte_stream = IterableIO(s.encode('utf-8') for s in string_data)

# Wrap with TextIOWrapper for text operations
text_stream = io.TextIOWrapper(byte_stream, encoding='utf-8')

print(f"First line: {text_stream.readline().strip()}") # Output: line1
print(f"Second line: {text_stream.readline().strip()}") # Output: line2
text_stream.close()

view raw JSON →