ScrapeGraph Python SDK

Common errors

• ModuleNotFoundError: No module named 'scrapegraph_py'

  cause The Python file containing your code has the same name as the package (`scrapegraph_py.py` or `scrapegraphai.py`), causing Python to import your local file instead of the installed library.
  fix Rename your Python script file to something different, e.g., `my_scraper.py`.

• ImportError: cannot import name 'OpenAI' from 'scrapegraphai.models'

  cause Attempting to import `OpenAI` directly from `scrapegraphai.models` when it's not exposed or has been moved/renamed in the current SDK version. The SDK typically handles LLM integration internally via the API key.
  fix Avoid direct imports of specific LLM classes like `OpenAI` from `scrapegraphai.models`. Instead, configure the desired LLM (e.g., OpenAI, Gemini) through the ScrapeGraphAI platform's API key. The `scrapegraph-py` SDK interacts with the ScrapeGraphAI API, which then manages the underlying LLMs.

• scrapegraph_py.exceptions.APIError: Invalid API key

  cause The provided API key is incorrect, expired, or missing. The `Client` could not authenticate with the ScrapeGraphAI service.
  fix Double-check your API key for typos. Ensure the `SGAI_API_KEY` environment variable is correctly set or that `api_key` is passed with a valid key to `Client(api_key='...')`. Obtain a new key from the ScrapeGraphAI Dashboard if necessary.

• scrapegraph_py.exceptions.APIError: Invalid URL format

  cause The `website_url` parameter passed to a scraping method (e.g., `smartscraper`) is malformed or not a valid URL.
  fix Verify that the `website_url` string is a complete and correctly formatted URL, including the scheme (e.g., 'https://example.com').

Warnings

• breaking The ScrapeGraph ecosystem has seen API changes, particularly with the transition to 'v2 API surface' in related projects like Scrapegraph-ai. While `scrapegraph-py` (the SDK) strives for stability, older code using `ScrapeGraphClient` functions (e.g., `smart_scraper(client, url, prompt)`) may need to be updated to the `Client` class methods (e.g., `client.smartscraper(website_url, user_prompt)`).
  Fix: Migrate usage from standalone functions like `smart_scraper(client, ...)` to methods on the `Client` instance, e.g., `client.smartscraper(...)`. Refer to the latest official documentation for current API patterns.
• gotcha Failing to set the `SGAI_API_KEY` environment variable or providing an invalid API key will result in authentication errors (HTTP 401 Unauthorized) or 'Insufficient credits' errors. The client will not be able to perform API calls.
  Fix: Ensure `SGAI_API_KEY` is set in your environment variables or passed directly to the `Client` constructor: `client = Client(api_key="your-api-key-here")`. Verify the API key's validity on the ScrapeGraphAI Dashboard.
• gotcha When using local LLMs with certain functionalities, a 'Model not found, using default token size (8192)' error or an `ImportError: Could not import transformers python package` might occur. This indicates issues with LLM configuration or missing dependencies.
  Fix: Ensure `transformers` is installed (`pip install transformers`). If configuring local LLMs, verify the `model_tokens` in your configuration dictionary is correctly set or remove it to use defaults. Consult the ScrapeGraphAI documentation for specific local LLM configurations.
• gotcha Running web scraping operations too frequently or without adhering to website policies (e.g., `robots.txt`, terms of service) can lead to IP blocking (HTTP 429 Too Many Requests) or other service unavailability errors (HTTP 500, 503).
  Fix: Implement rate limiting and delays between requests. Always check `robots.txt` and a website's terms of service before scraping. The SDK offers automatic retries for certain errors, but manual retry logic with backoff can also be implemented.

Install

• pip install scrapegraph-py Core Installation
• pip install scrapegraph-py[html] With HTML Validation (for website_html parameter)
• pip install scrapegraph-py[langchain] With Langchain Integration

Imports

• Client
  wrong:

  from scrapegraph_py import ScrapeGraphClient

  correct:

  from scrapegraph_py import Client

  The primary client class for interacting with the ScrapeGraphAI API. 'ScrapeGraphClient' might be from an older API version or another related library in the ecosystem.
• TimeRange

  from scrapegraph_py.models import TimeRange

  Used for filtering search results by date range in SearchScraper.

Quickstart

This quickstart demonstrates how to initialize the ScrapeGraph client using an API key (preferably from an environment variable) and then use the `smartscraper` service to extract structured data from a webpage. It utilizes a Pydantic `BaseModel` to define the desired output schema for robust data validation and type safety. The client should always be closed after use.

import os
from scrapegraph_py import Client
from pydantic import BaseModel, Field

# Set your ScrapeGraph AI API key
# It's recommended to set this as an environment variable: SGAI_API_KEY
# For quick testing, you can pass it directly or use os.environ.get
api_key = os.environ.get('SGAI_API_KEY', 'your_scrapegraph_api_key_here')

if not api_key or api_key == 'your_scrapegraph_api_key_here':
    print("Warning: Please set your SGAI_API_KEY environment variable or replace 'your_scrapegraph_api_key_here' with your actual API key.")
    exit()

client = Client(api_key=api_key)

class ArticleData(BaseModel):
    title: str = Field(description="The article title")
    author: str = Field(description="The author's name")
    publish_date: str = Field(description="Article publication date")
    content: str = Field(description="Main article content")

try:
    # Use SmartScraper to extract structured data from a webpage
    response = client.smartscraper(
        website_url="https://example.com/blog/article-example",
        user_prompt="Extract the article information",
        output_schema=ArticleData
    )

    print(f"Title: {response.title}")
    print(f"Author: {response.author}")
    print(f"Published: {response.publish_date}")
    print(f"Content snippet: {response.content[:100]}...")

finally:
    # Always close the client connection
    client.close()