{"library":"prophecy-libs","title":"Prophecy Python Libraries","description":"Prophecy Python Libraries (`prophecy-libs`) provides helper functions and utilities for Python code generated by the Prophecy data engineering platform. It facilitates the execution, configuration, and integration of Prophecy-generated data pipelines with Apache Spark. The library is actively maintained with frequent releases, typically accompanying platform updates.","language":"python","status":"active","last_verified":"Fri Apr 17","install":{"commands":["pip install prophecy-libs"],"cli":null},"imports":["from prophecy.config import ConfigStore","from prophecy.udf import UDFs","from prophecy.main import ProphecyApp"],"auth":{"required":false,"env_vars":[]},"quickstart":{"code":"import os\nfrom pyspark.sql import SparkSession\nfrom prophecy.udf import UDFs\n\n# This quickstart demonstrates how to initialize a SparkSession\n# and register Prophecy's User-Defined Functions (UDFs).\n# In a real Prophecy pipeline, this setup is usually handled automatically\n# by the generated pipeline entry point.\n\n# Ensure PySpark is installed and available in your environment.\n# E.g., `pip install pyspark==3.3.0` (or appropriate version based on prophecy-libs requirements)\n\ndef run_quickstart():\n    # Attempt to use a temporary directory for Spark warehouse for local testing\n    warehouse_dir = os.path.join(os.getcwd(), \"spark-warehouse\")\n    if not os.path.exists(warehouse_dir):\n        os.makedirs(warehouse_dir)\n\n    spark = SparkSession.builder \\\n        .appName(\"ProphecyLibQuickstart\") \\\n        .config(\"spark.sql.warehouse.dir\", warehouse_dir) \\\n        .master(\"local[*]\") \\\n        .getOrCreate()\n\n    try:\n        print(\"SparkSession initialized.\")\n\n        # Register Prophecy UDFs\n        UDFs.register_all_udfs(spark)\n        print(\"Prophecy UDFs registered successfully.\")\n\n        # Example: Using a simple UDF (assuming 'concat' or similar exists after registration)\n        # Note: Actual UDFs depend on the Prophecy project's definitions.\n        # This is a placeholder to show usage.\n        df = spark.createDataFrame([(\"hello\", \"world\")], [\"col1\", \"col2\"])\n        try:\n            df.createOrReplaceTempView(\"my_table\")\n            result = spark.sql(\"SELECT concat(col1, ' ', col2) as greeting FROM my_table\")\n            print(\"\\nExample UDF usage (if 'concat' is available via UDFs):\")\n            result.show()\n        except Exception as e:\n            print(f\"Could not demonstrate UDF usage (e.g., concat): {e}\")\n\n    except Exception as e:\n        print(f\"An error occurred during quickstart: {e}\")\n    finally:\n        spark.stop()\n        print(\"SparkSession stopped.\")\n\nif __name__ == \"__main__\":\n    run_quickstart()","lang":"python","description":"This quickstart demonstrates the essential setup of a SparkSession and how to register Prophecy's User-Defined Functions (UDFs). It highlights the core interaction pattern, though typical usage is within code generated and orchestrated by the Prophecy platform.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":null}