extract

Extract

Extract structured data from a page. Detect tables, pull JSON-LD, extract repeated DOM patterns. Produces findings on completeness, row counts, and schema consistency.

When to Use

• Extract tabular data from a page (pricing, specs, listings)
• Pull JSON-LD or structured data from <script> tags
• Detect repeated DOM patterns (cards, lists, grids)
• Assess data quality (missing columns, truncation, schema gaps)
• Extract data from async-loading pages
• Read visually-rendered data invisible to DOM (canvas, images, anti-scrape)

Protocol

Acquire -> Normalize -> Analyze with Evidence Mode.

1. Connect

connect_tab({ url: "https://target.com/data-page", background: true })

2. Acquire + Normalize

await smart.waitForNetworkIdle({ timeout: 10000 }); // wait for async content
const data = await smart.extractData();              // detectTables + extractTable + JSON-LD
const page = await smart.extractPage();              // meta and structured data context
// data.tables[] — selector, columns[], rows[], totalRows, truncated
// data.structuredData — JSON-LD objects from <script> tags
// page.gaps[] — what failed during extractPage

For deeper control, use individual methods:

const candidates = await smart.detectTables();           // scored table-like structures
const table = await smart.extractTable("table.pricing"); // specific table extraction

For visually-rendered data (canvas, images, anti-scrape sites):

const ocr = await ocrScreenshot({ fullPage: true });

3. Analyze — produce findings

for (const table of data.tables || []) {
  smart.finding({
    claim: `Found ${table.totalRows} rows in table "${table.selector}"`,
    evidence: [
      `columns: ${table.columns.map(c => c.name).join(', ')}`,
      `truncated: ${table.truncated}`
    ],
    sourceUrl: page.capture.url,
    confidence: table.truncated ? "medium" : "high",
    method: "extractData", dimension: "data-structure"
  });

  // Data quality — flag sparse columns
  const empty = table.columns.filter(c => c.fillRate < 0.5);
  if (empty.length) {
    smart.finding({
      claim: `${empty.length} columns in "${table.selector}" are >50% empty`,
      evidence: empty.map(c => `${c.name}: ${Math.round((1 - c.fillRate) * 100)}% empty`),
      sourceUrl: page.capture.url, confidence: "medium",
      method: "extractData", dimension: "data-quality"
    });
  }
}

if (data.structuredData?.length) {
  smart.finding({
    claim: `${data.structuredData.length} JSON-LD blocks found`,
    evidence: data.structuredData.map(s => `@type: ${s['@type'] || 'unknown'}`),
    sourceUrl: page.capture.url, confidence: "high",
    method: "extractData", dimension: "data-structure"
  });
}

return {
  findings: smart.findings(),
  tables: data.tables?.length || 0,
  structuredData: data.structuredData?.length || 0
};