DataCore Common

0.6.0 verified Fri May 01 auth: no python

Wrapper functions for PySpark, currently at version 0.6.0. Provides helper utilities to simplify Spark DataFrame operations. Release cadence is irregular.

pip install datacorecommon

Common errors

error ImportError: cannot import name 'str_to_date' from 'datacorecommon' ↓

cause Library not installed or incorrect import path.

fix

Run 'pip install datacorecommon' and use 'from datacorecommon import str_to_date'.

error ModuleNotFoundError: No module named 'pyspark' ↓

cause PySpark not installed.

fix

Install PySpark: 'pip install pyspark'.

Warnings

gotcha The library is still early stage (0.6.0). APIs may change without notice. ↓

fix Pin version in requirements and review changelog before upgrading.

gotcha Requires Python >=3.6 and a compatible PySpark installation. May not work with newer Spark versions. ↓

fix Ensure PySpark is installed and matched to your Spark cluster version.

Imports

str_to_date

wrong

from datacorecommon.utils import str_to_date

correct

from datacorecommon import str_to_date

Old import path; direct top-level import is correct.

get_spark_session

wrong

import datacorecommon; datacorecommon.get_spark_session()

correct

from datacorecommon import get_spark_session

Top-level import works, but without from may cause namespace confusion.

Quickstart

Create a Spark session and use str_to_date to convert string column to date.

from datacorecommon import get_spark_session, str_to_date

spark = get_spark_session()
df = spark.createDataFrame([('2023-01-01',)], ['date_str'])
df = str_to_date(df, 'date_str', 'yyyy-MM-dd')
df.show()