Chunking real company docs: what broke and why
Every RAG tutorial agrees: split documents on headings, keep chunks around 500–800 tokens, attach metadata. It works beautifully on documentation written by people who use headings. Then I pointed the pipeline at our actual Notion workspace.
What real documents look like
Three things broke immediately.
Giant tables. Our ops team keeps a Notion page that is one enormous table — hundreds of rows of supplier data. Heading-aware splitting saw a single block and produced one chunk far over the token limit, which then got truncated mid-row. Retrieval would confidently return the first third of a table and nothing else.
Pages that are all bullets. Engineers write meeting notes as flat bullet lists with no headings at all. The splitter fell back to one chunk per page, so a question about a decision from one meeting retrieved the entire meeting — or missed it, because the relevant bullet was diluted by forty unrelated ones.
Mixed French and English. We are a French company; docs switch language mid-page, sometimes mid-sentence. Chunks that straddled a language boundary embedded strangely and retrieved worse than either language alone.
What I tried
Three strategies, in order:
- Fixed windows (512 tokens, 15% overlap). The dumb baseline. Never catastrophic, never good — it happily cuts tables and sentences anywhere.
- Heading-aware, ~600 tokens, with metadata. Split on headings where they exist, subdivide oversized sections, and prepend page title and section path to every chunk. Tables get flattened row-by-row with the header repeated. This fixed the truncated-table failure completely.
- Thread-level chunking for conversational content. Meeting notes and anything chat-shaped get chunked per topic-thread instead of per heading, keeping each decision with its context.
How I measured it
I built a 30-question eval set from real questions colleagues had asked, each labeled with the document that contains the answer. The metric is plain hit-rate: is the right document in the top five retrieved chunks. It runs in CI; any change to the pipeline that drops hit-rate blocks the merge, exactly like a failing unit test.
Fixed windows scored 60%. Heading-aware with metadata: 83%. Adding thread-level chunking for notes: 90%.
The honest conclusion
I spent a weekend evaluating embedding models before any of this. Swapping models moved hit-rate by two or three points. Chunking strategy moved it by thirty. If your retrieval is bad, look at what you feed the embedder before you shop for a better one.