{"library":"sparkdantic","title":"Sparkdantic","type":"library","description":"Sparkdantic is a Python library that bridges Pydantic models with PySpark schemas. It allows developers to define data structures using Pydantic, then automatically generate equivalent `pyspark.sql.types.StructType` schemas for use in Spark DataFrames. This simplifies data validation and schema management across Python applications and Spark environments. The current version is 2.8.0, and it maintains an active release cadence, frequently updating for Pydantic and PySpark compatibility.","language":"python","status":"active","last_verified":"Fri May 22","install":{"commands":["pip install sparkdantic pyspark"],"cli":null},"imports":["from sparkdantic import create_spark_schema","from pydantic import BaseModel"],"auth":{"required":false,"env_vars":[]},"links":{"homepage":null,"github":null,"docs":null,"changelog":null,"pypi":"https://pypi.org/project/sparkdantic/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null},"quickstart":{"code":"from typing import Optional, List\nfrom pydantic import BaseModel\nfrom sparkdantic import create_spark_schema\nfrom pyspark.sql import SparkSession\n\n# Define your Pydantic model\nclass Product(BaseModel):\n    product_id: int\n    name: str\n    price: float\n    description: Optional[str] = None\n    tags: List[str] = []\n\n# Generate the Spark schema from the Pydantic model\nspark_schema = create_spark_schema(Product)\n\n# Print the generated Spark schema (useful for verification)\nprint(\"Generated Spark Schema:\")\nprint(spark_schema)\n\n# Optionally, use it with a Spark DataFrame\nspark = SparkSession.builder.appName(\"SparkdanticExample\").getOrCreate()\n\n# Create an empty DataFrame with the defined schema\ndf = spark.createDataFrame([], schema=spark_schema)\n\nprint(\"\\nDataFrame created with generated schema:\")\ndf.printSchema()\n\nspark.stop()","lang":"python","description":"This quickstart demonstrates how to define a Pydantic model and use `create_spark_schema` to generate a corresponding PySpark `StructType`. It then shows how to initialize a SparkSession and create an empty DataFrame using the generated schema, printing both the raw schema and the DataFrame's schema for verification.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":{"tag":null,"tag_description":null,"last_tested":"2026-05-22","installed_version":"2.8.0","pypi_latest":"2.8.0","is_stale":false,"summary":{"python_range":"3.10–3.9","success_rate":100,"avg_install_s":32.3,"avg_import_s":0.86,"wheel_type":"sdist"},"results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.72,"mem_mb":20,"disk_size":"518.2M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":32.8,"import_time_s":0.51,"mem_mb":20,"disk_size":"518M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":1.13,"mem_mb":21.7,"disk_size":"525.7M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":31.9,"import_time_s":0.95,"mem_mb":21.7,"disk_size":"525M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":1.25,"mem_mb":21.8,"disk_size":"514.4M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":31.4,"import_time_s":1.19,"mem_mb":21.8,"disk_size":"514M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.8,"mem_mb":19,"disk_size":"513.7M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":31.4,"import_time_s":0.81,"mem_mb":19,"disk_size":"513M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.66,"mem_mb":18.9,"disk_size":"496.6M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"sparkdantic","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":33.9,"import_time_s":0.64,"mem_mb":18.9,"disk_size":"496M"}]}}