ODPS Python SDK and data analysis framework

Warnings

• breaking The class `odps.accounts.AliyunAccount` was renamed to `odps.account.CloudAccount`. Code directly importing or referencing the old name will break.
  Fix: Update your imports from `from odps.accounts import AliyunAccount` to `from odps.account import CloudAccount`.
• breaking MaxCompute V4 signature is enabled by default, which may cause issues with services or environments that do not support it.
  Fix: If your service does not support V4 signatures, you might need to explicitly configure PyODPS to use an older signature version, if such an option is available and supported by your MaxCompute instance.
• breaking Decimal precision and scale checks at the client side have been tightened to align with MaxCompute server-side behavior. This might cause existing client-side checks to fail that previously passed.
  Fix: Review and adjust your data handling logic involving decimal types to ensure they conform to the stricter precision and scale rules.
• gotcha When using third-party packages in Python UDFs for MaxCompute, import statements for these packages must be placed *inside* the UDF's `evaluate` method (or similar processing method). Placing them at the module level will lead to runtime errors because the package is only available within the execution context on the MaxCompute server.
  Fix: Move `import` statements for third-party libraries into the `evaluate` or `process` method of your UDF class. Ensure the third-party package is uploaded as an archive resource to MaxCompute.
• gotcha PyODPS `execute_sql()` or `run_sql()` methods are primarily for DQL (Data Query Language) and DML (Data Manipulation Language). They may not correctly execute all SQL statement types, particularly DDL (Data Definition Language) commands like `CREATE TABLE` or complex API commands.
  Fix: For DDL operations (e.g., `CREATE TABLE`, `DROP TABLE`), use the specific PyODPS methods on the `ODPS` object (e.g., `o.create_table()`, `o.delete_table()`). For API commands, use corresponding PyODPS API methods.
• gotcha Downloading large datasets entirely to a local machine using PyODPS can lead to Out-Of-Memory (OOM) errors, especially when dealing with MaxCompute's distributed nature.
  Fix: It is highly recommended to offload heavy data processing and computations to the MaxCompute cluster by leveraging PyODPS DataFrame API or MaxCompute SQL. Only download aggregated or sampled results to your local environment.

Install

• pip install pyodps Basic installation
• pip install pyodps[full] Full installation with Jupyter support

Imports

• ODPS

  from odps import ODPS

  The core entry point for interacting with MaxCompute (ODPS) services.
• DataFrame

  from odps.df import DataFrame

  Used for the pandas-like DataFrame API for data analysis on MaxCompute.
• options

  from odps import options

  Global configuration options for PyODPS behavior, e.g., enabling interactive mode or schema support.

Quickstart

This quickstart demonstrates how to initialize the `ODPS` object using environment variables for credentials and then access a MaxCompute table (`dual`) to inspect its schema and read a few records. Replace placeholder values like 'your-project' and 'your-endpoint' with your actual MaxCompute project and endpoint. It's highly recommended to use environment variables for sensitive credentials.

import os
from odps import ODPS

# Ensure environment variables are set for security and best practice
# ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET
access_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', 'your-access-id')
secret_access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', 'your-secret-access-key')
project = os.environ.get('ODPS_PROJECT', 'your-project')
endpoint = os.environ.get('ODPS_ENDPOINT', 'your-endpoint')

# Initialize ODPS object
o = ODPS(access_id, secret_access_key, project=project, endpoint=endpoint)

# Get a table object
table = o.get_table('dual')

# Print table schema details
print(f"Table Name: {table.name}")
print(f"Table Schema: {table.table_schema}")
print("First 5 records:")
with table.open_reader() as reader:
    for record in reader.read(5):
        print(record)