extract-web-content

Extract web content

Get the useful content out of a web page — without the nav, ads, and markup — as Markdown, plain text, structured metadata, or custom fields.

Choose the right tool

You want…	Tool	Where the result lives
Readable Markdown (articles, docs)	microlink_markdown	microlink.data.markdown
Raw plain text	microlink_text	microlink.data.text
Title / description / author / image / logo / date	microlink_meta	microlink.data
Specific fields via CSS selectors	microlink_extract + data	microlink.data.<field>

All tools take a url. Always confirm microlink.status === "success" before using the payload.

Common flows

Summarize or ingest an article — get Markdown, then work with it:

{ "url": "https://example.com/blog/post" }

Call microlink_markdown, read microlink.data.markdown, and summarize/analyze it. Markdown preserves headings, links, and lists, which helps the model keep structure. Use microlink_text instead when you want the barest content for token efficiency.

Get link metadata (for previews, citations, or indexing) — call microlink_meta. Narrow the payload with the meta object to only the fields you need:

{ "url": "https://example.com", "meta": { "title": true, "author": true, "date": true } }

Scrape custom fields — call microlink_extract with a data map of CSS rules. Each key becomes a field on microlink.data:

{
  "url": "https://news.ycombinator.com",
  "data": {
    "headline": { "selector": ".titleline > a" },
    "points": { "selector": ".score", "type": "number" },
    "allLinks": { "selectorAll": ".titleline > a", "attr": "href" }
  }
}

See references/data-rules.md for the full CSS-rule syntax (selector, selectorAll, attr, type, evaluate, fallbacks, and nested data).

Tips & gotchas

• Prefer Markdown for LLM input. It keeps document structure at a low token cost; fall back to microlink_text only when structure doesn't matter.
• Only public URLs. Paywalled or auth-gated pages need headers/proxy (PRO).
• Stale content? Pass force: true to bypass the cache, or set ttl (e.g. "6h").
• JS-rendered pages: microlink_extract supports waitForSelector, scripts, and prerender if content loads late; markdown/text render the resolved page.
• Large pages: extract only the fields you need with data rules instead of pulling the whole document, to keep responses small.
• Free endpoint returns 429 at 50 requests/day — set MICROLINK_API_KEY for more.