Everyone tuning a RAG system obsesses over the model and the embeddings. The thing that quietly makes or breaks retrieval quality is humbler: how you chop your documents into pieces. That's chunking, and it deserves more attention than it gets.
Why chunking matters
RAG retrieves chunks, not whole documents. If your chunks are too big, a retrieved piece contains the answer buried in irrelevant text, diluting the signal. Too small, and a chunk lacks the context to be meaningful — the answer is split across pieces that don't get retrieved together. The chunk is the unit of retrieval, so its size and boundaries directly shape what the model sees.
Retrieval can only return what your chunks contain. Bad chunk boundaries mean the right answer never surfaces as a unit.
Strategies
- Fixed-size chunks (e.g., every 500 tokens) are simple but blindly cut mid-thought.
- Semantic / structural chunking respects natural boundaries — paragraphs, sections, headings — keeping coherent ideas together.
- Overlap between chunks prevents answers from being severed at boundaries.
- Hierarchical approaches store both small chunks (for precise matching) and larger parent context (for the model to reason over).
The practical takeaway
There's no universal best chunk size — it depends on your documents and queries. Dense technical docs, chat logs, and legal contracts all want different treatment. The move is to treat chunking as a tunable part of the system: try a few strategies, evaluate retrieval quality on real questions, and iterate. It's often the single highest-leverage fix for a RAG system that's "retrieving the wrong stuff" — cheaper and more effective than swapping the model.
