scrapy

Scrapy Best Practices

Scrapy is the definitive framework for high-performance web scraping in Python. This guide outlines the essential practices for building resilient, scalable, and ethical crawlers. Adhere to these guidelines to ensure your Scrapy projects are maintainable, efficient, and robust against the dynamic web landscape of 2025.

1. Code Organization and Structure

Maintain a clean, logical project structure. This enhances readability, testability, and scalability.

1.1. Standard Project Layout

Always use scrapy startproject to initialize your project. This sets up the recommended directory structure.

❌ BAD: Manually creating files and directories

# Don't do this
mkdir my_scraper
cd my_scraper
touch scrapy.cfg items.py spiders/__init__.py ...

✅ GOOD: Use the Scrapy CLI

scrapy startproject my_project_name
cd my_project_name

1.2. Item Definitions (items.py)

Define your data models clearly using scrapy.Item subclasses. Always include type hints for better IDE support and code clarity.

❌ BAD: Generic dictionaries or untyped Item fields

# items.py
import scrapy

class ProductItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    # No type hints, hard to know expected data type

✅ GOOD: scrapy.Item with Field and typing hints

# items.py
import scrapy
from scrapy.item import Field
from typing import Optional, List

class ProductItem(scrapy.Item):
    url: str = Field()
    title: Optional[str] = Field()
    price: Optional[float] = Field()
    description: Optional[str] = Field()
    image_urls: List[str] = Field()
    category: Optional[str] = Field()

1.3. Spiders

Keep spiders focused on crawling logic and initial data extraction. They should yield Request objects and Item objects.

❌ BAD: Complex data processing or storage logic in spiders

# spiders/bad_spider.py
import scrapy
from my_project_name.items import ProductItem

class BadSpider(scrapy.Spider):
    name = "bad_spider"
    start_urls = ["http://example.com"]

    def parse(self, response):
        item = ProductItem()
        item['title'] = response.css('h1::text').get()
        # ... complex cleaning and validation here ...
        # ... database insertion logic here ...
        yield item