Scrapy Playwright

Warnings

• breaking Python 3.8 support was dropped in `scrapy-playwright==0.0.44`. Users on Python 3.8 or older must upgrade their Python version to >=3.10 to use recent versions.
  Fix: Upgrade Python to 3.10 or newer. Ensure your `requires_python` is set correctly.
• breaking The import path for `HTTP11DownloadHandler` was updated in `scrapy-playwright==0.0.45`. This might affect projects with highly customized download handler setups or direct imports.
  Fix: Review and update any custom configurations or direct imports related to Scrapy's HTTP download handlers if they are dependent on `scrapy-playwright`'s internal path for this component.
• deprecated Positional argument handling for the function passed to the `PLAYWRIGHT_PROCESS_REQUEST_HEADERS` setting was deprecated in version 0.0.41. Arguments should now be handled by keyword.
  Fix: Update custom `PLAYWRIGHT_PROCESS_REQUEST_HEADERS` functions to accept keyword arguments instead of positional ones.
• gotcha Playwright requires browser binaries to be installed separately after the Python package. The `pip install playwright` command does not install these binaries by default. Without them, Playwright will not function.
  Fix: Run `playwright install` in your terminal after installing the Python package to download the necessary browser binaries (Chromium, Firefox, WebKit).
• gotcha Scrapy must be configured to use an asyncio-compatible Twisted reactor (e.g., `twisted.internet.asyncioreactor.AsyncioSelectorReactor`) for `scrapy-playwright` to work correctly. Failing to do so will lead to asynchronous request failures.
  Fix: Add `TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'` to your `settings.py` file.
• gotcha Playwright instances consume significant memory and CPU. Running many concurrent Playwright pages can exhaust system resources, leading to crashes or slow performance. This is especially critical when running non-headless browsers.
  Fix: Manage concurrency using Scrapy's `CONCURRENT_REQUESTS` and `CONCURRENT_REQUESTS_PER_DOMAIN` settings. Additionally, `PLAYWRIGHT_MAX_PAGES_PER_CONTEXT` can limit Playwright's parallel page usage. Always use `headless=True` in `PLAYWRIGHT_LAUNCH_OPTIONS` unless debugging visually.
• gotcha Playwright's `Page.route` and `Page.unroute` methods are used internally by `scrapy-playwright`. Directly using these methods in user code can interfere with the library's functionality and lead to unexpected behavior.
  Fix: Avoid directly calling `Page.route` or `Page.unroute` on Playwright page objects obtained via `response.meta['playwright_page']` unless you have a deep understanding of the internal workings and are prepared for potential conflicts.

Install

• pip install scrapy-playwright playwright Install package
• playwright install Install browser binaries

Imports

• ScrapyPlaywrightDownloadHandler

  from scrapy_playwright.handler import ScrapyPlaywrightDownloadHandler

  Used in settings.py to enable Playwright for HTTP/HTTPS requests.
• PageMethod

  from scrapy_playwright.page import PageMethod

  Used to define actions (e.g., click, wait_for_selector) within a Playwright request's meta field.
• PlaywrightRequest

  from scrapy_playwright.request import PlaywrightRequest

  While Request objects are usually from 'scrapy', some examples might mistakenly use 'page' or an old path for Playwright-specific request objects. The standard approach is to use `scrapy.Request` with `meta={'playwright': True}`.

Quickstart

To use Scrapy Playwright, you must configure your `settings.py` file to include the `ScrapyPlaywrightDownloadHandler` and set the `TWISTED_REACTOR` for asyncio compatibility. Then, within your spider, create `scrapy.Request` objects with `meta={'playwright': True}`. You can also define page interactions using `playwright_page_methods` with `PageMethod` objects to wait for elements or perform actions before parsing.

import scrapy
from scrapy_playwright.page import PageMethod

# settings.py configuration (add these to your project's settings.py)
# DOWNLOAD_HANDLERS = {
#     "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
#     "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
# }
# TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
# PLAYWRIGHT_BROWSER_TYPE = "chromium" # or 'firefox', 'webkit'
# PLAYWRIGHT_LAUNCH_OPTIONS = {
#     "headless": True, # Set to False for visual debugging
#     "timeout": 60000 # 60 seconds
# }

class MySpider(scrapy.Spider):
    name = "my_spider"
    start_urls = ["https://quotes.toscrape.com/js/"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_selector", "div.quote"),
                        # PageMethod("screenshot", path="screenshot.png", full_page=True),
                    ],
                },
                callback=self.parse_quotes,
            )

    async def parse_quotes(self, response):
        # The response is now a PlaywrightResponse object with rendered content
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield scrapy.Request(response.urljoin(next_page), meta={'playwright': True, 'playwright_page_methods': [PageMethod("wait_for_selector", "div.quote")]}, callback=self.parse_quotes)