token-aware-chunking

Token-Aware Semantic Chunking Protocol (Boundary-Aligned RAG Slicing)

Overview

When indexing documentation or codebases for vector search and RAG retrieval, naive splitters slice text by fixed character counts (text[i:i+2000]).

Fixed-character chunking causes severe retrieval degradations:

1. Broken Code Blocks: Splits a TypeScript interface or Python function midway through its body, creating unparseable syntax fragments.
2. Mid-Word Token Clipping: Slices words across token boundaries, corrupting embedding vector representations.
3. Embedding Model Ceiling Exceedance: A character count that translates to 8,250 tokens gets silently truncated by an embedding model with an 8,192-token ceiling.

The Token-Aware Semantic Chunking Protocol measures chunk size strictly using the target tokenizer (tiktoken / BPE) and splits text recursively along semantic boundaries (Markdown Headers, AST Function Blocks, Double Newlines).

---

Fixed-Character Slicing vs. Token-Aware Semantic Chunking

┌─────────────────────────────────────────────────────────────┐
│                 Text Chunking Mechanics                     │
│                                                             │
│  Fixed Character Slicing (`len(text) == 2000`):             │
│  • Chunk 1 ends: `function calculateTotal(price: num`       │
│  • Chunk 2 starts: `ber, tax: number) { return price + ...` │
│  ↳ Syntax broken across 2 chunks! Vector embedding corrupted│
│                                                             │
│  Token-Aware Semantic Slicing (512 Tokens / AST Boundary):  │
│  • Chunk 1: Complete `calculateTotal` function + docstring  │
│  • Chunk 2: Complete `processPayment` function              │
│  ↳ 100% Valid code syntax, exact 512-token budget adherence │
└─────────────────────────────────────────────────────────────┘

---

The 4-Tier Semantic Split Hierarchy

When partitioning text into token-bounded chunks, search for split delimiters in descending priority: