people love blaming the model. the answer is wrong, so the model is wrong. swap the model. bump the temperature down. try a bigger context window. write a sterner system prompt. none of that helps if the retriever handed the model a paragraph that starts mid-sentence and ends mid-table.
chunking is boring, so teams treat it like plumbing. they pick a token size, maybe add overlap, ship it, and move on to the part that looks like ai. then the demos start failing in ways that feel mysterious. the model "ignores" the source. the model "hallucinates" a number that was two chunks away. the model "can't follow" a procedure that was split across a heading boundary. you can spend a week on prompt surgery and never find the leak, because the leak is upstream.
i have watched this pattern enough times that i now assume chunking first. not always, but often enough that it saves calendar time. if the retrieved text would confuse a careful human reader, it will confuse a model. that is not a deep insight. it just gets ignored because chunking does not show up in launch decks.
what bad chunks look like in production
the failures rarely arrive labeled "chunking." they arrive as product complaints.
someone asks for the refund window and gets an answer from a marketing page that mentions refunds in passing, while the actual policy lives in a numbered list that got cut after item two. someone asks how to rotate a key and gets step 1 and step 4, because steps 2 and 3 lived in the next chunk and lost the race. someone asks for a definition and gets a glossary entry that starts with "see also" and never includes the definition itself.
these look like reasoning failures. they feel like the model is being creative. sometimes it is. more often it is completing a broken local picture with whatever prior it has. that is what language models do when the local evidence is incomplete. if you only inspect the final answer, you will keep upgrading models. if you inspect the retrieved chunks, you will see the mess.
a useful habit: for every bad answer, print the exact context window the model saw. not the document titles. the text. read it cold. if you would not trust yourself to answer from that paste, stop tuning generation.
fixed windows are a default, not a strategy
most pipelines start with "split every n tokens with some overlap." that is fine as a scaffold. it is a bad final design for anything with structure.
documents are not soup. they have headings, lists, tables, code blocks, callouts, and footnotes. a fixed window does not care. it will bisect a table row. it will detach a heading from the section it names. it will leave a bullet list without its intro sentence, so the bullets look like freestanding claims. overlap helps a little. overlap also duplicates noise and can promote the wrong neighbor when similarity search is noisy.
structure-aware chunking is not glamorous. it means walking the document tree, keeping headings with their bodies, keeping list items with their lead-in, and refusing to emit a chunk that is only a caption or only a page number. it means deciding what a "unit of meaning" is for your corpus, which is a product decision pretending to be an engineering one.
if your corpus is mostly support articles, chunk by section under a stable heading path. if it is mostly api docs, keep endpoint blocks intact. if it is mostly pdf reports with tables, admit that pdf extraction is the real boss fight and fix that before you argue about embedding models.
the orphan heading problem
orphan headings are my favorite silent killer. you get a chunk that is mostly whitespace and a title like "rate limits," then the next chunk has the actual numbers. retrieval returns the heading chunk because the query contains "rate limits." the model sees a title and invents plausible limits, or hedges into uselessness. either way, users blame the model.
the fix is dull: never index a heading alone. fold short headings into the following body, or store heading path as metadata and require body text above a minimum length. if a section is short, keep the whole section as one chunk even if that makes your sizes uneven. uneven chunk sizes are fine. confident wrong answers are not.
tables and the fantasy of linear text
tables die under naive chunking. a row needs its column headers. a cell like "30" means nothing without "days" and "enterprise plan." if your extractor flattens a table into a reading-order string and then slices by tokens, you will retrieve fragments that look numeric and authoritative while missing the label that makes them true.
when people say the model "made up a sla," check whether the sla row was retrieved without its header. models are good at filling blanks. your job is to stop handing them blanks that look like answers.
practical approach: serialize tables into self-contained records. each row becomes a small statement that repeats the headers. yes, that costs tokens. yes, that is usually cheaper than a support escalation.
overlap theater
overlap gets treated like insurance. fifty tokens of overlap, problem solved. sometimes overlap saves you. sometimes it just means the same broken boundary appears twice, once at the end of chunk a and once at the start of chunk b, and both get retrieved, which crowds out a better section from elsewhere in the doc.
overlap is a patch for a boundary policy you do not trust. if you need heavy overlap to make answers work, your units are wrong. prefer fewer, cleaner units over a puree with a sliding window.
also watch for near-duplicate chunks after overlap. embedding indexes will happily return three slightly different cuts of the same paragraph. your reranker then spends its budget on clones. the model gets three copies of half a thought and still misses the sentence that actually answers the question.
metadata amnesia
a chunk without provenance is a rumor. "in section 4.2 of the 2024 enterprise policy" is different from a free-floating paragraph that happens to mention enterprises. if you strip titles, version labels, product names, and effective dates during chunking, retrieval can surface text from the wrong product line or an old policy revision. the model may answer fluently from that text. fluency is not correctness.
keep a breadcrumb with every chunk: document title, section path, version, and whatever product dimension you use in routing. put the breadcrumb in the embedded text or in filters you actually apply. i have seen teams store beautiful metadata and then never filter on it. that is how you get last year's pricing in this year's chatbot.
evaluation that cannot see chunks
if your eval only scores final answers, you will misdiagnose systematically. add a retrieval inspection step. for a failing question, record whether the gold span was present in any retrieved chunk, whether it was split, and whether a distractor chunk outranked it. that triage tells you if you have a chunking problem, a ranking problem, or a generation problem.
teams skip this because it is annoying to maintain gold spans. the alternative is arguing about models in a meeting while the same section boundary keeps deleting the answer. pick your annoyance.
a cheap version works: ten real failing tickets, manual paste of retrieved context, a human labels "answerable from context" yes or no. if the answerable rate is low, stop touching prompts.
symptoms checklist, without the checklist cosplay
when quality drops after a corpus update, look at what changed in segmentation before you look at model routing. new pdfs, new exporters, a docs site redesign, a confluence migration: all of these change boundaries. embeddings did not suddenly get worse on tuesday. your chunks did.
when answers cite the right document and still get the detail wrong, suspect intra-document chunking. when answers cite the wrong document, suspect metadata and routing. when answers are vague, suspect that retrieved chunks are headings, summaries, or nav chrome. when answers invent numbers, suspect tables and lists that lost their labels.
none of this requires a research paper. it requires reading the inputs.
what to do instead of model churn
start with a target unit for each major doc type. optimize for intact units, not for a pretty length histogram. enforce minimum body length. attach section paths. serialize tables on purpose. run the ten-ticket paste test after every corpus or parser change.
then argue about embedding models. embeddings can only rank what you gave them. if you gave them shredded manuals, they will retrieve shredded manuals with great semantic precision.
models fail constantly. a surprising fraction of "model problems" in rag stacks are still chunking mistakes with good pr. fix the cuts. the model often looks smarter when you stop feeding it sentence fragments and calling it inference.