Koalas: pandas API on Apache Spark

Warnings

• breaking Koalas as a standalone library is deprecated. All its functionality has been officially integrated into PySpark as 'pandas API on Spark' starting with Apache Spark 3.2. Users are strongly advised to migrate to PySpark directly.
  Fix: For Apache Spark 3.2 and later, use `import pyspark.pandas as ps` instead of `import databricks.koalas as ks`. Migrate existing Koalas code to use `pyspark.pandas`.
• breaking The default plotting backend for Koalas switched from Matplotlib to Plotly in version 1.7.0. This can change the visual output and require different plotting options.
  Fix: Review existing plotting code. If you prefer Matplotlib, you might need to explicitly set the backend, e.g., `ks.set_option('plotting.backend', 'matplotlib')` or adapt to Plotly's capabilities.
• gotcha Koalas historically had different behavior than pandas regarding unnamed Series. Prior to v1.2.0, Koalas would automatically name a Series '0' if no name was specified, unlike pandas which allows a truly unnamed Series. This was fixed in v1.2.0 to align with pandas.
  Fix: Ensure your code explicitly handles Series names if consistency is critical, or upgrade to Koalas 1.2.0+ for pandas-like unnamed Series behavior.
• gotcha Compatibility with specific pandas versions can introduce subtle bugs. For example, Koalas 1.8.2 addressed an issue with `_builtin_table` import in `groupby.apply` that affected pandas versions 1.3.0 and above.
  Fix: Always check release notes for specific pandas version compatibility. Keep Koalas updated to the latest available version if you are using recent pandas versions.
• gotcha Early versions of Koalas (pre-1.5.0) had limited or inconsistent support for complex Index operations (e.g., chained arithmetic operations), sometimes raising `AssertionError`.
  Fix: Upgrade to Koalas 1.5.0 or later for improved Index operation support, or refactor complex index manipulations to simpler steps in older versions.

Install

• pip install koalas Install Koalas

Imports

• Koalas DataFrame/Series

  import databricks.koalas as ks

  The official package namespace is `databricks.koalas`.
• Koalas DataFrame/Series (from pandas)

  import databricks.koalas as ks
  import pandas as pd

  While Koalas mimics pandas, it is a separate library and should be imported under its own alias, typically `ks`.

Quickstart

This quickstart demonstrates how to create a Koalas DataFrame from a pandas DataFrame, perform a basic operation (calculate mean), and convert it back to a pandas DataFrame. Ensure you have PySpark configured in your environment for this to run against a Spark session.

import databricks.koalas as ks
import pandas as pd

# Create a pandas DataFrame
pdf = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Convert to a Koalas DataFrame
kdf = ks.DataFrame(pdf)

print("Koalas DataFrame head:")
print(kdf.head())

print("Mean of column 'A':", kdf['A'].mean())

# Convert back to a pandas DataFrame
pdf_result = kdf.to_pandas()
print("\nConverted back to pandas DataFrame:")
print(pdf_result)