chunking

Chunking

Use this when feeding documents into an LLM context window or a vector store. Xberg chunks two ways: inline during extraction (chunks land on each document's chunks field), or standalone via the chunk command for text you already have. Sizing is character-based by default, or token-based when a tokenizer model is supplied.

Inline during extraction

Turn on chunking with --chunk and the chunks appear on the structured result under chunks:

# 1000-char chunks, 200-char overlap (defaults when --chunk is on)
xberg extract report.pdf --chunk --format json | jq '.chunks | length'

# Explicit size + overlap
xberg extract report.pdf --chunk --chunk-size 1500 --chunk-overlap 300 --format json

Overlap must be smaller than chunk size — the CLI rejects --chunk-overlap >= --chunk-size. When you set only --chunk-overlap against an existing config, an overlap that exceeds the size is clamped to chunk_size / 4.

Standalone chunk command

Chunk text you already have, from --text or stdin. Output defaults to JSON:

# From a flag
xberg chunk --text "long document text ..." --chunk-size 800 --chunk-overlap 100

# From stdin (pipe extracted content straight in)
xberg extract notes.md | xberg chunk --chunk-size 500 --format json

JSON output carries chunks (array of strings), chunk_count, the resolved config (max_characters, overlap, chunker_type), and input_size_bytes. Use --format text for a human-readable dump with --- chunk N --- separators.

Note: in the JSON output, chunker_type is rendered capitalized ("Text", "Markdown", "Yaml", "Semantic") because it is emitted via Rust's Debug formatting, whereas the --chunker-type input flag is lowercase (text, markdown, yaml, semantic). Lowercase the value before comparing if you parse it back.

Chunker types

--chunker-type selects the splitting strategy (standalone chunk command):

Type	Behavior
text	Default. Plain character-window splitting with overlap.
markdown	Markdown-aware — splits on structure (headings, blocks) where possible.
yaml	YAML-aware splitting for structured config/data documents.
semantic	Topic-boundary splitting driven by --topic-threshold (0.0–1.0, default 0.75).