chunking-for-llms

Syntax-aware chunking for LLMs

Splitting code on a fixed byte or line count cuts functions in half and strips context. ts-pack process <file> --chunk-size <bytes> splits on syntactic boundaries (whole functions, classes, blocks) so each chunk is a coherent unit, and emits them in the JSON chunks array.

Quick recipe

# ~2 KB chunks aligned to syntax boundaries
ts-pack process src/app.ts --chunk-size 2000

--chunk-size is a maximum size in bytes. The splitter packs whole syntactic units up to that bound; an oversized single construct becomes its own chunk rather than being cut. Chunks are added to the normal process JSON output under chunks.

Picking a size

• Match the downstream model's token budget. A rough rule: bytes ÷ 4 ≈ tokens for code, so --chunk-size 4000 is on the order of ~1k tokens.
• Larger chunks preserve more local context but fit fewer per request.
• Leave headroom for the prompt, the surrounding messages, and the response — do not size chunks to the full context window.

Combining with extraction

Chunking composes with the other process features, so you can attach structure metadata to each request:

ts-pack process src/service.py --structure --chunk-size 3000 \
  | jq '{chunks: (.chunks | length), functions: (.structure | length)}'

Chunk output

chunks is a list of code-chunk objects in the process JSON. Each chunk carries its source text plus span information (line/byte offsets), so you can cite or re-locate a chunk back in the original file. Iterate the array to feed an LLM one coherent unit at a time:

ts-pack process big_module.py --chunk-size 2500 \
  | jq -c '.chunks[]'

SDK equivalent

The SDK exposes chunking through the process config: set the chunk_max_size field (in bytes) on ProcessConfig — the same value the CLI's --chunk-size flag sets. ProcessConfig is a frozen dataclass, so pass it to the constructor:

from tree_sitter_language_pack import process, ProcessConfig

config = ProcessConfig("python", chunk_max_size=2500)
result = process(source_code, config)
for chunk in result.chunks:          # ProcessResult is an object, not a dict
    send_to_llm(chunk.content)