extracting-keywords

Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword extraction, language detection, and vector embeddings. Keywords and language detection ride along with extraction and land on the result; embeddings are produced by a dedicated embed command.

Keywords (YAKE / RAKE)

Keyword extraction is configured via the [keywords] config block (or inline JSON) — there is no single --keywords CLI flag. When enabled, extracted keywords appear on result.extracted_keywords (extractedKeywords in Node.js; the CLI JSON field is extracted_keywords). Two algorithms are available:

• YAKE ("yake") — statistical, unsupervised single-document extraction. Good general default.
• RAKE ("rake") — co-occurrence / phrase-based. Favors multi-word key phrases.

Feature-gated: keyword extraction requires the CLI to be built with the keywords-yake and/or keywords-rake Cargo features (both are in the default/full build). If the CLI was built without them, the [keywords] config block is silently ignored — result.extracted_keywords simply stays empty rather than erroring. The "yake" algorithm needs keywords-yake; "rake" needs keywords-rake.

Enable via inline JSON on the CLI:

xberg extract paper.pdf --format json \
  --config-json '{"keywords":{"algorithm":"yake","max_keywords":15,"language":"en"}}' \
  | jq '.extracted_keywords'

Or in a config file:

[keywords]
algorithm = "rake"       # "yake" or "rake"
max_keywords = 10        # default 10
min_score = 0.0          # filter below this score (normalized 0.0-1.0 for both algorithms)
ngram_range = [1, 3]     # unigrams..trigrams (default); config-file only
language = "en"          # stopword language; omit to skip stopword filtering

xberg extract report.pdf --config xberg.toml --format json | jq '.extracted_keywords'

Field notes:

• max_keywords caps how many keywords are returned (default 10).
• min_score filters low-scoring keywords. Both YAKE and RAKE normalize their scores to the 0.0-1.0 range with higher-is-better, so min_score retains keywords with score >= min_score identically for either algorithm.
• ngram_range is [min, max]: [1,1] unigrams only, [1,2] adds bigrams, [1,3] (default) adds trigrams. Config-file only — it is not a field on the language bindings' KeywordConfig.
• language enables stopword filtering for that language; omit it to disable stopword filtering entirely.