Scrapy

Warnings

• breaking TWISTED_REACTOR default changed to asyncio (AsyncioSelectorReactor) in 2.13. Existing projects that relied on the default reactor being None may behave differently. Projects with incompatible Twisted code that assumed the default reactor could silently break.
  Fix: Explicitly set TWISTED_REACTOR in settings.py if you need a specific reactor. To restore old behavior: TWISTED_REACTOR = None. New projects use asyncio by default which is correct.
• breaking start_requests() (sync) replaced by start() (async) in 2.13. The iteration behavior changed: start requests now run continuously rather than stopping when the scheduler has pending requests. This can cause different crawl ordering and memory behavior on large crawls.
  Fix: Override start() instead of start_requests() in new spiders. For existing spiders: start_requests() still works but its iteration behavior changed. See 'Delaying start request iteration' in docs to restore previous behavior.
• breaking Python 3.9 dropped in Scrapy 2.13. Minimum is now Python 3.10.
  Fix: Pin Scrapy<2.13 for Python 3.9 environments.
• gotcha response.css() and response.xpath() return SelectorList, not strings. Forgetting .get() or .getall() returns a SelectorList object, not the text. A common source of silent data bugs.
  Fix: Use .get() for the first match (returns str or None), .getall() for all matches (returns list of str). Example: response.css('h1::text').get() not response.css('h1::text').
• gotcha return in a parse callback instead of yield causes items/requests to be silently dropped. parse() must be a generator (use yield) not return a list.
  Fix: Replace return [item1, item2] with yield item1; yield item2. Or return a generator expression. Scrapy 2.13 added a warning for this (WARN_ON_GENERATOR_RETURN_VALUE setting).
• gotcha Running scrapy crawl outside a Scrapy project directory raises ConfigError. The scrapy CLI requires a scrapy.cfg file in the current or parent directory.
  Fix: Always run scrapy commands from inside a project directory (where scrapy.cfg is). Create a project first: scrapy startproject myproject.

Install

• pip install Scrapy Standard (note capital S in package name)
• scrapy startproject myproject Create new project

Imports

• Spider.start

  import scrapy
  
  class MySpider(scrapy.Spider):
      name = 'myspider'
      start_urls = ['https://example.com']
  
      # New async start() method (2.13+) — preferred over start_requests()
      async def start(self):
          for url in self.start_urls:
              yield scrapy.Request(url, callback=self.parse)
  
      def parse(self, response):
          yield {'title': response.css('title::text').get()}

  start_requests() (sync) was replaced by start() (async) in Scrapy 2.13. start_requests() still works but start() is the new preferred interface. Custom start_requests() overrides still function but cannot yield items directly.

Quickstart

Basic spider. Run with: scrapy crawl quotes -o output.json

import scrapy

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = ['https://quotes.toscrape.com']

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        # Follow pagination
        next_page = response.css('li.next a::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)