scraper-models

---

description: This rule provides the description of the possible scraper types that can be created. globs: **/*.py

Scraper types

Here's a list of possible scraper types and ther data structures

E-commerce PLP

• The structure of the items in a PLP scraper is the following: website_name, extraction_date, product_code, item_url, full_price, price, currency, image_url, brand, product_category1, product_category2, product_category3, product_name
• When asked to create an e-commerce PLP scraper, set items.py file and in the scraper ouput accordingly to the data structure.
• PLP pages are the product list pages in an e-commerce, also called catalogue pages. In a e-commerce PLP scraper, the scraper should crawl all the product catalog without entering the pages with product details.
• The scraper will usually start from a home page.

E-commerce PDP

• The strucutre of the items in a PDP scraper is the following: website_name, extraction_date, product_code, item_url, full_price, price, currency, image_url, brand, product_category1, product_category2, product_category3, product_name, product_description, product_size, product_color, additional_info
• When asked to create an e-commerce PDP scraper, set items.py file and in the scraper ouput accordingly to the data structure.
• PDP pages are product detail pages, the final leaf of an e-commerce website. An e-commerce PDP scraper will have in input a list of PDP pages and won't need to crawl the website further.

How to fill the scraper fields with values

• When asked to map a field of a data structure to the information contained in the HTML, use the following rules:
  • website_name: this is a fixed value per each scraper, usually the website's name in upper case. If in doubt, ask to the operator
  • extraction_date: fixed value for the whole execution, YYYY-MM-DD format. Use datetime library
  • product_code: code that identifies every single product on the website
  • item_url: URL of the page containing the details of the product. If PDP data structure, it corresponds to response.url
  • full_price: price before the discounts. If there's no discount on the item, it's the selling price.
  • price: final selling price after the discounts. If no discount is on the website, it's the selling price.
  • currency: ISO3 Code for currency, fixed value for a whole scraper. Detect the currency from the HTML and use the ISO Code to populate the field
  • brand: brand or producer of the product sold on the website
  • product_category1: first level or product categorization, usually the first level of the breadcrumb of the page, if any.
  • product_category2: second level or product categorization, usually the second level of the breadcrumb of the page, if any.
  • product_category3: third level or product categorization, usually the third level of the breadcrumb of the page, if any.
  • product_name: name of the product as shown on the pages
• In any case and in any field of a scraper, do not hardcode any value but always find a selector to get the correct one.
• Always print in output every field of the scraper, even if it's empty.