beautifulsoup4

beautifulsoup4 Best Practices

Beautiful Soup 4 (BS4) is the definitive library for parsing static HTML/XML. Use it as a robust component within a well-structured scraping pipeline.

1. Installation & Parser Selection

Always install beautifulsoup4 alongside lxml for optimal performance. lxml is significantly faster than Python's built-in html.parser. html5lib is a useful fallback for extremely malformed HTML.

pip install beautifulsoup4 lxml requests html5lib

✅ GOOD: Explicitly specify lxml as the parser.

from bs4 import BeautifulSoup

# Always use lxml for speed
soup = BeautifulSoup(html_content, 'lxml')

❌ BAD: Relying on default html.parser or html5lib unnecessarily.

# Slower, less efficient
soup = BeautifulSoup(html_content, 'html.parser')
# Only use html5lib if lxml fails on malformed HTML
soup = BeautifulSoup(html_content, 'html5lib')

2. Ethical Scraping Fundamentals

Respect robots.txt, implement rate limiting, and use a User-Agent. This is non-negotiable for responsible scraping.

✅ GOOD: Respect robots.txt and use a User-Agent with delays.

import requests
import time
from typing import Dict

def fetch_page_ethically(url: str, headers: Dict[str, str], delay_seconds: float = 2.0) -> str:
    """Fetches HTML content with ethical considerations."""
    # In a real app, check robots.txt programmatically
    # e.g., using 'robotexclusionrulesparser' library
    print(f"Fetching {url}...")
    response = requests.get(url, headers=headers, timeout=10)
    response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
    time.sleep(delay_seconds) # Rate limiting
    return response.text

# Example usage
HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 MyScraper/1.0'
}
# html_content = fetch_page_ethically("https://example.com/products", HEADERS)

❌ BAD: Blindly hitting endpoints without identification or delays.

# Will likely get blocked or cause issues
response = requests.get("https://example.com/products")
html_content = response.text