using-web-scraping

Web Scraping Skill — Chrome (Playwright) + DuckDuckGo

A privacy-minded, agent-facing web-scraping skill that uses headless Chrome (Playwright/Puppeteer) and DuckDuckGo for search. Focuses on: reliable navigation, extracting structured text, obeying robots.txt, and rate-limiting.

When to use

• Collect public webpage content for summarization, metadata extraction, or link discovery.
• Use DuckDuckGo for queries when you want a privacy-respecting search source.
• NOT for bypassing paywalls, scraping private/logged-in content, or violating Terms of Service.

Safety & etiquette

• Always check and respect /robots.txt before scraping a site.
• Rate-limit requests (default: 1 request/sec) and use polite User-Agent strings.
• Avoid executing arbitrary user-provided JavaScript on scraped pages.
• Only scrape public content; if login is required, return login_required instead of attempting to bypass.

Capabilities

• Search DuckDuckGo and return top-N result links.
• Visit result pages in headless Chrome and extract title, meta description, main text (or best-effort article text), and canonical URL.
• Return results as structured JSON for downstream consumption.

Examples

Node.js (Playwright)

const { chromium } = require('playwright');

async function ddgSearchAndScrape(query) {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage({ userAgent: 'open-skills-bot/1.0' });

  // DuckDuckGo search
  await page.goto('https://duckduckgo.com/');
  await page.fill('input[name="q"]', query);
  await page.keyboard.press('Enter');
  await page.waitForSelector('.result__title a');

  // collect top result URL
  const href = await page.getAttribute('.result__title a', 'href');
  if (!href) { await browser.close(); return []; }

  // visit result and extract
  await page.goto(href, { waitUntil: 'domcontentloaded' });
  const title = await page.title();
  const description = await page.locator('meta[name="description"]').getAttribute('content').catch(() => null);
  const article = await page.locator('article, main, #content').first().innerText().catch(() => null);

  await browser.close();
  return [{ url: href, title, description, text: article }];
}

// usage
// ddgSearchAndScrape('open-source agent runtimes').then(console.log);