Splitting every document at 512 tokens is not a RAG strategy
How you split a document changes what the model can retrieve and what it loses. Start with the document structure and the questions people actually ask.
Chunking defines the evidence unit visible to retrieval. A fixed 512-token window is only a starting point; headings, tables, code, and question scope need different boundaries.
Preserve document structure first
Split on sections, paragraphs, lists, tables, and code blocks and retain heading path and page metadata. Token-split only oversized units, keeping headers with rows and signatures with implementations.
Measure retrieval with real questions
Label supporting passages for real questions and compare context recall, precision, and rank across chunking variants. Large chunks add noise; small chunks lose context. Vector similarity alone is insufficient.
Handle special content explicitly
Represent tables as rows with headers, split code by symbols or syntax, and quality-check OCR for scanned PDFs. Retrieval may expand to neighbors, but record the final source range.
Pre-release checks
- Citations map to source locations
- Structured content is not arbitrarily split
- Retrieval evaluation supports the choice
Sources
Related
Writing, extraction, long reasoning, and high-volume support do not need the same model. Cost, latency, privacy, and reliability rarely point to one name.
GPUs, operations, inference tuning, and downtime all land on the same ledger. Self-hosting only makes sense after you have done the math on volume and people.