ivbeg/newsworker

Advanced news feeds extractor and finder library. Helps to automatically extract news from websites without RSS/ATOM feeds

87Stars on the repository
11Mods indexed here, across every type
12d agoLast push, which is what freshness is scored on
MITLicence, which decides whether bodies are shown

analyze

01

ivbeg/newsworker

Command

Runs the dynamic heuristics once and distils them into a portable YAML parsing spec. Feeding that spec back into extract --spec skips analysis and runs deterministic selectors.

87 12d ago A 5 tokens original MIT

batch

02

ivbeg/newsworker

Command

Extracts feeds from a list of pages concurrently, writing one file per URL.

87 12d ago A 5 tokens original MIT

cache

03

ivbeg/newsworker

Command

Inspects or clears the spec and content caches. See Settings.

87 12d ago A 5 tokens original MIT

extract

04

ivbeg/newsworker

Command

Extracts news items from an HTML page and renders them in the chosen format.

87 12d ago A 5 tokens original MIT

index

05

ivbeg/newsworker

Command

Index of newsworker CLI commands.

87 12d ago A 6 tokens original MIT

parsedate

06

ivbeg/newsworker

Command

A debugging helper that shows how qddate interprets a date string.

87 12d ago A 6 tokens original MIT

scan

07

ivbeg/newsworker

Command

Scans a page for already-published RSS/Atom/JSON Feed declarations (via autodiscovery links, feed icons, link heuristics, WebSub, and optional sitemaps) and reports them.

87 12d ago A 5 tokens original MIT

serve

08

ivbeg/newsworker

Command

Runs a lightweight local HTTP server (Python standard library, no extra dependencies) that turns any page URL into a feed on demand over GET.

87 12d ago A 5 tokens original MIT

spec

09

ivbeg/newsworker

Command

Validate a parsing spec without fetching its source URL.

87 12d ago A 5 tokens original MIT

watch

10

ivbeg/newsworker

Command

Polls a page on an interval, tracks which items it has already seen (in a SQLite store under the cache dir), and emits or delivers only new items.

87 12d ago A 5 tokens original MIT