tinsel: PySpark schema generator

0.3.0 · maintenance · verified Thu Apr 16

Tinsel is a lightweight Python library designed to simplify PySpark DataFrame schema generation. It allows users to define complex PySpark schemas using familiar Python native types like `NamedTuple` and `dataclasses`, removing the need for verbose PySpark schema DSLs. The library is small, fast, and provides type shims for some Python types that might not have direct Spark equivalents. The current version is 0.3.0, with the last public update in September 2018, indicating a maintenance-level release cadence.

Common errors

```
ModuleNotFoundError: No module named 'tinsel'
```
cause The `tinsel` library is not installed in the current Python environment.

fix Run `pip install tinsel` to install the library.
```
TypeError: 'StructType' object is not callable
```
cause Attempting to call the result of `transform(YourClass)` as if it were a function, or misusing the generated schema object.

fix Ensure the output of `transform()` is assigned to a variable (e.g., `schema = transform(YourClass)`) and then passed to PySpark's `createDataFrame` using the `schema=` keyword argument (e.g., `spark.createDataFrame(data, schema=schema)`).
```
AttributeError: 'module' object has no attribute 'struct' or 'transform'
```
cause This usually means `struct` or `transform` was imported incorrectly, or the `tinsel` package itself is not properly installed or is shadowed by another module.

fix Verify that `from tinsel import struct, transform` is used. Check your Python environment for any conflicting packages named `tinsel` or issues with the installation.

Warnings

gotcha The `tinsel` library has not seen updates since September 2018. While its core functionality remains valid, it might not be compatible with the absolute latest features or changes in very recent PySpark versions or Python language constructs.
Fix: Thoroughly test `tinsel` generated schemas with your specific PySpark and Python versions. Consider manual schema definition for highly complex or cutting-edge PySpark features if issues arise.
gotcha Tinsel handles nullable fields and provides 'type shims' for certain Python types that don't have direct PySpark equivalents (e.g., `long` or `short`). Users should be aware of how these types are mapped to avoid unexpected schema interpretations.
Fix: Review the generated PySpark schema (via `df.printSchema()`) carefully to confirm that types and nullability match expectations. Refer to the `tinsel` source for explicit type mapping details if ambiguities occur.

Install

pip install tinsel Install latest version

Imports

struct
wrong:
```
from tinsel.schema import struct
```
correct:
```
from tinsel import struct
```
The `struct` decorator is directly available from the top-level `tinsel` package.
transform
wrong:
```
import tinsel.transform
```
correct:
```
from tinsel import transform
```
The `transform` function is a direct import from the `tinsel` package.

Quickstart

This quickstart demonstrates how to define a PySpark schema using Tinsel with Python's `dataclasses` and `NamedTuple`. It then converts this definition into a `StructType` compatible with PySpark and creates a DataFrame with sample data.

from dataclasses import dataclass
from typing import NamedTuple, Optional, Dict, List
from tinsel import struct, transform
from pyspark.sql import SparkSession

# Define nested schema using dataclass
@struct
@dataclass
class UserInfo:
    hobby: List[str]
    last_seen: Optional[int]
    pet_ages: Dict[str, int]

# Define root schema using NamedTuple
@struct
class User(NamedTuple):
    login: str
    age: int
    active: bool
    info: Optional[UserInfo]

# Transform the Python class into a PySpark schema
spark_schema = transform(User)

# Prepare sample data matching the defined structure
data = [
    User(
        login="Ben",
        age=18,
        active=False,
        info=None
    ),
    User(
        login="Tom",
        age=32,
        active=True,
        info=UserInfo(
            hobby=["pets", "flowers"],
            last_seen=16,
            pet_ages={
                "Jack": 2,
                "Sunshine": 6
            }
        )
    )
]

# Initialize SparkSession
spark = SparkSession.builder.master('local').appName("TinselQuickstart").getOrCreate()

# Create DataFrame using the generated schema and data
df = spark.createDataFrame(data=data, schema=spark_schema)
df.printSchema()
df.show(truncate=False)

spark.stop()

view raw JSON →