google-search-crawler CLAUDE.md

CLAUDE.md - Google Search Crawler

Project Philosophy

This project embodies the principle that elegant code is inevitable code. Every component, every function, every line exists not because it can, but because it must. We build tools that feel like natural extensions of the developer's mind.

Vision

Google Search Crawler is a Python library that makes web scraping feel intuitive. It transforms the complexity of HTTP requests, cookie management, pagination, and HTML parsing into a simple, beautiful generator pattern that any developer can understand in seconds.

Core Principles

1. Simplicity Through Elegance

The entire API surfaces through a single function: search(). Everything else is either a convenience wrapper or an internal implementation detail. Users shouldn't need to understand cookies, user agents, or URL encoding to get results.

for result in search('python programming', stop=10):
    print(result['title'], result['url'])

That's it. That's the entire interface. Beautiful.

2. Generator-First Architecture

We use Python generators not for performance (though that's a bonus), but because they match the mental model perfectly. Search results are inherently sequential and potentially infinite. A generator captures this truth elegantly.

3. Fail Gracefully, Always

The web is unpredictable. Google changes their HTML. Networks fail. Rate limits trigger. Our code handles every failure gracefully, logs it properly, and recovers when possible. Users should never see a cryptic stack trace.

4. Respect The Web

We implement rate limiting not because we have to, but because it's the right thing to do. We randomize user agents to avoid detection, but we also pause between requests to be respectful. Good tools should make it easy to do the right thing.

Architecture

The Three Layers

┌─────────────────────────────────────┐
│   API Layer (search, search_images) │
│   - User-facing functions            │
│   - Parameter validation             │
│   - Convenience wrappers             │
└─────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────┐
│   Core Engine (search generator)     │
│   - Pagination logic                 │
│   - Result parsing                   │
│   - Deduplication                    │
└─────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────┐
│   Foundation (get_page, filter)      │
│   - HTTP requests                    │
│   - Cookie management                │
│   - HTML parsing                     │
└─────────────────────────────────────┘