Scrapy

Warnings

• breaking Scrapy has progressively dropped support for older Python versions. Scrapy 2.12.0 dropped Python 3.8 support, and Scrapy 2.14.0 dropped Python 3.9 and PyPy 3.10. Ensure your environment meets the `requires_python >=3.10` requirement.
  Fix: Upgrade to Python 3.10 or newer. Use a virtual environment for isolated Scrapy installations.
• breaking The `start_requests()` (synchronous) method for yielding initial requests has been replaced by `start()` (asynchronous) in Scrapy 2.13.0, which is now the preferred way. While `start_urls` remains a shortcut, direct asynchronous operations for initial requests should use `async def start(self)`.
  Fix: Migrate `def start_requests(self)` to `async def start(self)` if you are performing asynchronous operations (e.g., database calls) to generate your initial requests.
• breaking The asyncio reactor is now enabled by default starting from Scrapy 2.13.0. This might affect applications with existing Twisted-specific reactor setups or require updates for custom spider middlewares that do not explicitly support asynchronous output, which may now log warnings.
  Fix: Review and update custom middlewares to support asynchronous spider output by defining `process_spider_output` as an asynchronous generator or implementing `process_spider_output_async`.
• breaking Scrapy 2.14.2 includes a security fix where values from the `Referrer-Policy` header of HTTP responses are no longer executed as Python callables. Additionally, 301 redirects of POST requests are now converted into GET requests, aligning with the HTTP standard.
  Fix: Do not rely on `Referrer-Policy` header values being executed as code. Be aware that POST requests resulting in 301 redirects will now be re-sent as GET requests.
• deprecated `scrapy.utils.defer.maybeDeferred_coro()` and other related `scrapy.utils.defer` functions (e.g., `mustbe_deferred`, `defer_succeed`, `defer_fail`) are deprecated in Scrapy 2.14.1. Users are encouraged to use direct Twisted functions or coroutines.
  Fix: Replace calls to `scrapy.utils.defer` functions with their `twisted.internet.defer` equivalents or appropriate coroutine patterns. For `maybeDeferred_coro()`, consider `twisted.internet.defer.maybeDeferred` if staying with Deferreds.
• gotcha Request and Response objects now define `__slots__`, meaning you cannot assign arbitrary attributes directly (e.g., `response.foo = 1`). Attempting to do so will raise an `AttributeError`.
  Fix: Store per-request/response data in the `request.meta` or `request.cb_kwargs` mappings instead of attaching new attributes to the objects.
• gotcha In Scrapy 2.13.3, the default project template changed the values for `DOWNLOAD_DELAY` (from 0 to 1 second) and `CONCURRENT_REQUESTS_PER_DOMAIN` (from 8 to 1) to promote more polite crawling. New projects will inherit these slower defaults.
  Fix: Adjust `DOWNLOAD_DELAY` and `CONCURRENT_REQUESTS_PER_DOMAIN` in your `settings.py` if you require higher concurrency or a faster crawl rate for your specific use case.

Install

• pip install scrapy Install stable version

Imports

• scrapy

  import scrapy

• Spider

  from scrapy import Spider

  Importing Spider directly from scrapy is generally preferred and cleaner than from scrapy.spider.
• Request

  from scrapy import Request

• Item

  from scrapy import Item

• Field

  from scrapy.item import Field

• CrawlSpider

  from scrapy.spiders import CrawlSpider

• LinkExtractor

  from scrapy.linkextractors import LinkExtractor

• AsyncCrawlerProcess

  from scrapy.crawler import AsyncCrawlerProcess

  Used for running Scrapy from scripts. AsyncCrawlerProcess returns coroutines, while CrawlerProcess returns Deferred objects.

Quickstart

This quickstart demonstrates a basic Scrapy spider that crawls the 'quotes.toscrape.com' website, specifically the 'humor' tag. It extracts the author and text of each quote, then follows the 'Next Page' link to continue crawling. The `start_urls` attribute defines the initial URLs, and the `parse` method handles the response, extracting data and scheduling new requests using `response.follow` for pagination.

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        "https://quotes.toscrape.com/tag/humor/",
    ]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "author": quote.xpath("span/small/text()").get(),
                "text": quote.css("span.text::text").get(),
            }

        next_page = response.css('li.next a::attr("href")').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

# To run this spider, save it as a .py file (e.g., quotes_spider.py) and execute:
# scrapy runspider quotes_spider.py -o quotes.jsonl