Scrapfly Python SDK

Common errors

• AttributeError: 'ScrapeApiResponse' object has no attribute 'selector'

  cause You are trying to use the `.selector` property on a `ScrapeApiResponse` object for HTML parsing, but the necessary optional dependencies (`parsel` or `scrapy`) have not been installed.
  fix Install the `parser` extra: `pip install "scrapfly-sdk[parser]"`. If you are using Scrapy, install `pip install "scrapfly-sdk[scrapy]"` instead.

• scrapfly.errors.ScrapflyError: Invalid API key (HTTP 401 Unauthorized)

  cause The Scrapfly API key provided to `ScrapflyClient` is either missing, incorrect, expired, or has insufficient permissions.
  fix Verify your API key from your Scrapfly dashboard (https://scrapfly.io/dashboard) and ensure it's correctly passed during client initialization, ideally from an environment variable.

• scrapfly.errors.ScrapflyError: Too Many Requests (HTTP 429)

  cause Your Scrapfly account has exceeded its allocated request rate limit or concurrent request limit for the given time period.
  fix Implement exponential backoff and retry logic in your scraping code. Review your Scrapfly plan limits or consider upgrading your plan if sustained higher rates are needed.

• Scraped content is empty, incomplete, or shows an anti-bot page (no SDK error, but unexpected content).

  cause The target website's anti-bot protection mechanisms successfully identified and blocked the scraping request, or the page content requires JavaScript rendering.
  fix In your `ScrapeConfig`, enable `render_js=True`, `asp=True` (Anti-Scraping Protection), and potentially specify a `proxy_pool='public_residential_pool'` and `country='us'` (or relevant target country).

Warnings

• gotcha Accessing the `ScrapeApiResponse.selector` property for built-in HTML parsing requires installing either `parsel` or `scrapy` as an optional dependency (e.g., `pip install "scrapfly-sdk[parser]"`). Without these, attempting to use `.selector` will result in an `AttributeError`.
  Fix: Install the `parser` or `scrapy` extra: `pip install "scrapfly-sdk[parser]"`
• gotcha Hardcoding your Scrapfly API key directly in your code is insecure and inflexible. It's best practice to retrieve it from environment variables or a secure configuration system.
  Fix: Use `os.environ.get('SCRAPFLY_API_KEY', 'default_or_error_key')` to load your API key, and set the environment variable `SCRAPFLY_API_KEY`.
• gotcha Scrapfly API errors (e.g., HTTP 400, 401, 429, 5xx) are encapsulated by `scrapfly.errors.ScrapflyError` subclasses. Incorrect handling or misinterpretation of these can lead to brittle scrapers. Consult the official Scrapfly error documentation for detailed explanations and suggested remedies.
  Fix: Implement robust `try-except` blocks for `ScrapflyError` and its subclasses. Refer to the Scrapfly API troubleshooting guide for specific error codes and their meanings.
• breaking While not explicitly documented as a breaking change for the Python SDK `0.10.0` specifically, other Scrapfly SDKs (e.g., TypeScript SDK v0.6.9) have undergone parameter renames (e.g., `ephemeral_template` to `extraction_ephemeral_template` in the Extraction API). Always review the official Changelog or release notes for potential API parameter changes when upgrading, especially across minor or major versions.
  Fix: Before upgrading, check the `scrapfly-sdk` GitHub repository's release notes or the official Scrapfly documentation's changelog for any API parameter renames or deprecations.

Install

• pip install scrapfly-sdk Basic Installation
• pip install "scrapfly-sdk[all]" Full Installation with all extras

Imports

• ScrapflyClient

  from scrapfly import ScrapflyClient

• ScrapeConfig

  from scrapfly import ScrapeConfig

Quickstart

This quickstart demonstrates how to initialize the Scrapfly client and perform a basic scrape request to a test page. It shows how to enable JavaScript rendering and specify a proxy country. Remember to replace 'YOUR_SCRAPFLY_API_KEY' with your actual key or set the SCRAPFLY_API_KEY environment variable. For HTML parsing with `.selector`, ensure `parsel` or `scrapy` is installed as an extra dependency.

import os
from scrapfly import ScrapflyClient, ScrapeConfig

SCRAPFLY_API_KEY = os.environ.get('SCRAPFLY_API_KEY', 'YOUR_SCRAPFLY_API_KEY')

async def main():
    client = ScrapflyClient(key=SCRAPFLY_API_KEY)
    try:
        result = await client.scrape(ScrapeConfig(url='https://web-scraping.dev/product/1', render_js=True, country='us'))
        print(f"Status: {result.status_code}")
        print(f"Content length: {len(result.content)} bytes")
        # If 'parsel' or 'scrapy' is installed, you can use .selector
        # print(f"Product Title: {result.selector.css('h3::text').get()}")
    except Exception as e:
        print(f"An error occurred: {e}")
    finally:
        await client.close()

if __name__ == '__main__':
    import asyncio
    asyncio.run(main())