PyORC

0.11.0 · active · verified Thu Apr 16

PyORC is a Python module designed for efficiently reading and writing data in the Apache ORC (Optimized Row Columnar) file format. It provides high-performance access to ORC files, commonly used in big data ecosystems like Apache Hive, Spark, and Flink. The current version is 0.11.0, and the library maintains an active development schedule with several releases per year.

Common errors

```
TypeError: object of type '_io.BufferedReader' has no len()
```
cause Attempting to pass a file-like object directly to the `path` argument of `pyorc.Reader` after version 0.10.0. The `path` argument now expects a string or `pathlib.Path`.

fix Use the `file_like_object` keyword argument for file-like objects: `reader = pyorc.Reader(file_like_object=f_obj)`.
```
pyorc.errors.PyorcDataError: The given data is not compatible with the current schema
```
cause The Python tuple/list passed to `writer.write()` does not match the structure or data types specified in the `TypeDescription` provided to the `pyorc.Writer`.

fix Review your `TypeDescription` and the data being written. Ensure column order, data types, and (for decimals) precision/scale are correctly aligned. For example, if schema expects `string`, don't pass `int`.
```
pyorc.errors.ParseError: Malformed ORC file
```
cause The ORC file being read is corrupted, truncated, or not a valid ORC file. This can also occur if the file was written with incompatible compression or encoding options.

fix Verify the integrity of the ORC file. Ensure it's not truncated. If writing, try different `compression` or `stripe_size` options. If reading, ensure the file is indeed an ORC file.

Warnings

breaking The `pyorc.Reader` constructor's behavior changed significantly for file-like objects. Previously, you could pass a file-like object directly as the `path` argument. Now, `path` is strictly for string or `pathlib.Path` objects. For file-like objects, you must pass them to the `file_like_object` keyword argument.
Fix: If you are opening a file-like object (e.g., from `open()`, `io.BytesIO`), use `pyorc.Reader(file_like_object=f_obj)`. For a file path, use `pyorc.Reader('path/to/file.orc')`.
breaking The `TypeKind` enum was moved directly under the `pyorc` module. Additionally, default values for some `Column` attributes (e.g., `tzinfo`) were changed.
Fix: Update imports from `from pyorc.enums import TypeKind` to `from pyorc import TypeKind`. Review `pyorc.Column` usage if relying on default `tzinfo` or other schema defaults.
gotcha When writing `datetime` objects to ORC `timestamp` columns, it is highly recommended to use timezone-aware `datetime` objects (e.g., using `datetime.timezone.utc` or `pytz`). Writing naive `datetime` objects can lead to ambiguous or incorrect time interpretations in downstream systems.
Fix: Ensure all `datetime` objects passed to `pyorc.Writer` have an explicit timezone, preferably UTC. Example: `datetime.datetime(YYYY, M, D, H, M, S, tzinfo=datetime.timezone.utc)`.
gotcha PyORC enforces strict schema matching during writing. If the Python data types or structure do not align precisely with the defined ORC `TypeDescription`, a `PyorcDataError` will be raised. This includes precision and scale for `decimal` types.
Fix: Carefully define your `TypeDescription` to match the data you intend to write. Ensure `decimal` values have correct precision/scale. Validate your data against the schema before writing.

Install

pip install pyorc Basic installation
pip install 'pyorc[dataframe]' With Pandas DataFrame support

Imports

Reader
```
from pyorc import Reader
```
Writer
```
from pyorc import Writer
```
TypeDescription
```
from pyorc import TypeDescription
```
TypeKind
wrong:
```
from pyorc.enums import TypeKind
```
correct:
```
from pyorc import TypeKind
```
TypeKind was moved directly under the pyorc module in version 0.9.0

Quickstart

This quickstart demonstrates how to define an ORC schema, write data into an ORC file, and then read the data back using `pyorc.Writer` and `pyorc.Reader`. It includes examples for `int`, `string`, `decimal`, and `timestamp` types, ensuring timezone awareness for `datetime` objects.

import pyorc
import os
import datetime
import decimal

# Define a schema for demonstration
schema_str = "struct<id:int,name:string,value:decimal(10,2),timestamp:timestamp>"
schema = pyorc.TypeDescription.from_string(schema_str)

file_path = "example.orc"

# --- Writing an ORC file ---
print(f"Writing to {file_path}")
with open(file_path, "wb") as f:
    with pyorc.Writer(f, schema) as writer:
        writer.write((1, "Alice", decimal.Decimal("10.50"), datetime.datetime(2023, 1, 1, 10, 0, 0, tzinfo=datetime.timezone.utc)))
        writer.write((2, "Bob", decimal.Decimal("20.75"), datetime.datetime(2023, 1, 2, 11, 30, 0, tzinfo=datetime.timezone.utc)))
        writer.write((3, "Charlie", decimal.Decimal("30.00"), datetime.datetime(2023, 1, 3, 12, 0, 0, tzinfo=datetime.timezone.utc)))

print(f"Successfully wrote {file_path}")

# --- Reading an ORC file ---
print(f"Reading from {file_path}")
with open(file_path, "rb") as f:
    # For file-like objects, pass them directly to the Reader constructor (v0.10.0+)
    reader = pyorc.Reader(f)
    print("Schema:", reader.schema)
    print("Rows:")
    for row in reader:
        print(row)

# Clean up
if os.path.exists(file_path):
    os.remove(file_path)
    print(f"Cleaned up {file_path}")

view raw JSON →