agentcore-browser-web-scraping

AgentCore Browser Web Scraping

Amazon Bedrock AgentCore Browser gives you a managed cloud Chromium — no crawler fleet, no resident containers, per-session billing, natural isolation. This skill captures a production scraping architecture built on it: an LLM agent decides what to do on the page, but only through fixed code primitives it can parameterize, never arbitrary scripts. Login state lives in Browser Profiles that users populate once through a live-view session, so credentials never transit the conversation.

Collector (Python, worker thread)
  ├─ browser_session(region, profile_configuration=..., proxy_configuration=...)
  │      └─ StartBrowserSession → generate_ws_headers() (SigV4)
  │             └─ playwright chromium.connect_over_cdp(ws_url, headers)
  └─ Strands Agent (LLM) with 6 fixed tools:
        navigate / scroll_to_bottom / click_load_more /
        get_page_text / screenshot / extract_by_selector

When to Apply

Reference this skill when:

• Scraping dynamic, JS-rendered, or lazy-loading pages (social feeds, forums, review sites) with AgentCore Browser + Playwright.
• Scraping sites that require login (X/Twitter, Reddit, Instagram) and you need reusable login state that never leaves AWS.
• Scrapes return 0 items on pages that clearly have content — usually a scroll-container or login-wall misdiagnosis, both covered here.
• Google SSO inside the cloud browser silently fails (third-party cookies).
• The target site blocks AWS egress IPs and you need an external proxy.

Not for: general AgentCore Browser service overview or session APIs — see the aws-agentic-ai skill's Browser service docs for that.

How It Works

1. Session, connection, and the LLM-driven extraction loop

Connect Playwright to the managed browser over a SigV4-signed CDP WebSocket, then let an LLM agent drive a bounded tool loop. Key decisions:

• LLM decides, code executes. A hand-written script per site doesn't scale across arbitrary DOMs; raw LLM-generated JS is an injection surface. The middle path: six fixed tools, the model only supplies parameters.
• extract_by_selector: a constant JS extraction template evaluated with the model's CSS selectors passed as data arguments — zero string concatenation, zero eval. Preserves item boundaries and attributes that a flat innerText dump loses.
• screenshot (downscaled JPEG returned as an image tool-result) lets the model see the page and distinguish "genuinely empty" from login wall / CAPTCHA / cookie banner before declaring failure.
• Adaptive step budget: scale the agent's max tool-steps (and the session timeout) with the requested record count — lazy feeds yield 10–20 items per scroll, so a fixed small budget silently under-collects.
• Scrolling that actually works: document.body.scrollHeight is 0 on flex layouts (YouTube) — scroll document.scrollingElement instead, and also iterate inner overflow panels; comments often live in an overlay that window scrolling never touches.
• SSRF guard: HTTPS-only, no IP literals, private ranges and IMDS blocked for the hostname and every resolved IP — re-validated inside the navigate tool on every call, so the model can't pivot mid-session.