website-analysis

description: This rule provides a step by step guide to analyze a website and its code, in order to write a better Scrapy scraper. globs: **/*.py

How to perform a website analysis before writing the code of a Scrapy scraper

• From the conversation, please identify the type of Scrapy spider that should be built (e.g. E-commerce PLP, E-commerce PDP, etc.) and confirm it by writing it in the chat
• Fetch the home page HTML and store it in the absolute path of the project, naming the file homepage.html, while store the cookies in the cookies.txt file
• Using the MCP tool strip_css, strip the CSS from the file homepage.html and save the new version on the file homepage_stripped.html
• Read the file cookies.txt and look for anti-bot software traces, like Akamai, Datadome, PerimeterX and so on. For every cookie name, check if it can be referred to an anti bot solution.
• If Akamai is found between the anti-bots, when creating your scraper you'll need to add the scrapy_impersonate package in the execution to make it work
• If Datadome or Kasada are found betwen the anti-bots, stop the process
• If you're asked to create an e-commerce PLP scraper, follow these steps:
  • Ask for a product category URL and fetch its HTML, saving it in the absolute path of the project, with the name category.html. Store the cookies in the cookies_category.txt file.
  • Using the MCP tool strip_css, strip the CSS from the file category.html and save the new version on the file category_stripped.html
  • Read the file homepage_stripped.html and look for any well formatted JSON you can use to get all the product categories URL, included the one just passed. Look for schema.org but also common frameworks like Next.js. Be careful that schema.org JSON does not include the full price without discount of a product, so you'll need to look for it elsewhere.
  • Read the file category_stripped.html and look for any well formatted JSON you can use to read the product details of every product on the category page. If there's any, save this JSON in a file called catalog.json
• If you're asled to create a PDP scraper, follow these steps:
  • Ask for a product page and fetch its HTML, saving it in the absolute path of the project, with the name product.html. Store the cookies in the cookies_product.txt file.
  • Using the MCP tool strip_css, strip the CSS from the file product.html and save the new version on the file product_stripped.html
  • Read the file product_stripped.html and look for any well formatted JSON you can use to read the product details of every product on the category page. If there's any, save this JSON in a file called product.json. Look for schema.org but also common frameworks like Next.js. Be careful that schema.org JSON does not include the full price without discount of a product, so you'll need to look for it elsewhere.