sodapy: Python client for Socrata Open Data API

Warnings

• breaking The `sodapy` library is officially unmaintained as of August 31, 2022. No new features or bug fixes will be added. While existing functionality still works, users should proceed with caution and consider the lack of ongoing support for new projects.
  Fix: Be aware of the unmaintained status; evaluate if the existing functionality meets your long-term needs without requiring future updates or bug fixes. For data management operations (creating/transforming data), consider `socrata-py` which targets the Socrata Data Management API.
• gotcha Queries executed without an application token will be subjected to strict throttling limits by the Socrata API. This can lead to slower responses or request failures for extensive data retrieval.
  Fix: Always use an application token when initializing the `Socrata` client, especially for frequent or large queries. You can obtain one from the Socrata website after creating an account. Pass it as the `app_token` argument: `Socrata(domain, app_token='YOUR_APP_TOKEN')`.
• gotcha The Socrata Open Data API (SODA) and `sodapy` are primarily for reading and directly writing to datasets. For write operations that involve data transformations or use the Socrata Data Management Experience (e.g., creating datasets through the UI), the Socrata Data Management API should be used.
  Fix: Understand the distinction between the SODA API and the Data Management API. If your task involves complex data transformations or is related to the data management user interface, investigate the `socrata-py` SDK which is designed for the Data Management API.
• gotcha Socrata API calls have a default timeout of 10 seconds. For large datasets or slow connections, this can result in `Readtimeout error` exceptions, preventing full data retrieval.
  Fix: Increase the timeout limit for the `Socrata` client by setting the `timeout` parameter during initialization or by updating the `client.timeout` attribute. Example: `client = Socrata(..., timeout=60)` or `client.timeout = 60`.
• gotcha SODA APIs are paged and typically return a maximum of 50,000 records per request. Without proper pagination, you might only retrieve a subset of the available data.
  Fix: Use the `$limit` and `$offset` parameters in your `client.get()` calls to request subsequent pages of data. To retrieve all records, you might need to make multiple paginated requests or use the `client.get_all()` method if available and suitable for your dataset.

Install

• pip install sodapy Install with pip

Imports

• Socrata

  from sodapy import Socrata

Quickstart

Initializes a Socrata client, retrieves the first 5 records from a public dataset (e.g., NYC 311 Service Requests), and fetches its metadata. It demonstrates setting up the client with an optional application token and credentials, and performing a basic data retrieval. Environment variables are used for secure credential handling.

import os
from sodapy import Socrata

# Get credentials from environment variables or provide directly
APP_TOKEN = os.environ.get('SOCRATA_APP_TOKEN', None) # Recommended for higher rate limits
USERNAME = os.environ.get('SOCRATA_USERNAME', None) # Only required for creating/modifying data
PASSWORD = os.environ.get('SOCRATA_PASSWORD', None) # Only required for creating/modifying data

# Example: Connect to a public dataset (e.g., NYC Open Data - 311 Service Requests)
# Replace 'data.cityofnewyork.us' with your Socrata domain
# Replace 'erm2-nwe9' with your dataset identifier
domain = 'data.cityofnewyork.us'
dataset_identifier = 'erm2-nwe9'

with Socrata(domain, APP_TOKEN, username=USERNAME, password=PASSWORD) as client:
    # Increase timeout for large datasets if needed
    # client.timeout = 50

    # Example: Retrieve the first 5 records
    print(f"Retrieving the first 5 records from {dataset_identifier} on {domain}...")
    results = client.get(dataset_identifier, limit=5)

    # Results are returned as a list of dictionaries
    for item in results:
        print(item)

    print(f"\nRetrieved {len(results)} records.")

    # Example: Retrieve metadata for the dataset
    print(f"\nRetrieving metadata for {dataset_identifier}...")
    metadata = client.get_metadata(dataset_identifier)
    print(f"Dataset Name: {metadata.get('name')}")
    print(f"Description: {metadata.get('description', '')[:100]}...")