scrapy-quick-spider

Scrapy Quick Spider

Builds a single-file, runnable Scrapy spider for ecommerce category crawling using Zyte API's AI extraction. No project setup. Just a .py file the user can run immediately.

---

When to use this skill

User provides a category/listing URL and wants all products from it scraped. Output is a self-contained Python script saved to their workspace.

---

What you need from the user

• Entry URL — the category or product listing page (e.g. https://shop.com/category/widgets)
• Output format (optional) — default to jsonl

That's it. No detail page URL needed — the spider discovers product URLs from the listing page automatically.

---

How it works

The spider uses two Zyte API AI extraction types, chained together:

1. productNavigation on the category page → returns product URLs + pagination
2. product on each product URL → returns structured product data

No selectors. No schema definition. No user confirmation step. Just generate and save.

Read references/zyte-extraction-fields.md to understand the exact fields returned by each extraction type before writing the spider.

---

Spider architecture

Use AsyncCrawlerProcess to run from a script (no project needed):

from scrapy import Spider, Request
from scrapy.crawler import AsyncCrawlerProcess

Spider flow

start()
  └─► parse_navigation()       # productNavigation extraction
        ├─► parse_navigation()  # follow nextPage if present
        └─► parse_product()     # product extraction for each item URL

AsyncCrawlerProcess

Use AsyncCrawlerProcess instead of CrawlerProcess — it manages the asyncio event loop internally. Do not wrap in asyncio.run() or use await process.crawl(). Call process.crawl(Spider) (sync) then process.start() to run. Wrapping in asyncio.run() causes RuntimeError: Event loop stopped before Future completed on exit.

All spider methods must be async.

Zyte API integration