most rag postmortems start with the model. then the prompt. then the chunk size. metadata shows up late, if at all, usually as a shrug: "we can add filters later." later is when you discover that half your top hits are the wrong product version, the wrong audience, or a draft that was never meant to answer customers.
metadata is not a nice-to-have on top of embeddings. it is the part of retrieval that lets you refuse bad candidates before the model ever sees them. when you skip it at ingest, you are betting that cosine similarity alone will know the difference between a 2022 policy and a 2025 policy that share a lot of vocabulary. that bet fails quietly. the answer still looks fluent. the citation still points at a real document. the user still gets the wrong number.
i used to treat metadata as a cleanup task. store the text, ship the demo, backfill fields when product asks for filters. that order is backwards. the fields you skip are the ones you will need the first time someone asks a versioned question, or a locale-specific question, or a question that should never pull from marketing copy.
what "we will filter later" actually costs
the pain shows up as mysterious relevance.
a support agent asks about refund timing and gets a blog post that mentions refunds as a brand virtue. a developer asks about an api field and gets docs for the previous major version, because the wording barely changed and the embedding space does not care about your release calendar. a compliance person asks for the current retention rule and gets an internal draft that was never published, sitting in the same index as the live policy because nobody tagged status.
none of these are embedding failures in the interesting sense. the vectors did their job. similar text ranked high. the system lacked the boring labels that would have dropped those candidates before ranking got clever.
people respond by adding more context to the query, or by writing a reranker prompt that says "prefer official sources." that can help a little. it is also a tax you pay forever for not storing a boolean that said audience=customer or status=published at ingest time. models are bad at reconstructing organizational intent from prose alone. your pipeline already knew the intent when it ingested the file. it just threw that knowledge away.
there is a second cost that is harder to see: debugging gets slower. when every bad answer could be "wrong chunk," "wrong model," or "wrong doc entirely," you burn hours reading text that should never have been eligible. with even a thin metadata layer, you can ask a sharper question: among eligible docs, did ranking fail? that is a different investigation from "why did we cite a deprecated runbook."
the fields that keep paying rent
you do not need a warehouse of attributes on day one. you need the ones that match how your users already disambiguate documents when they search by hand.
source type is the obvious one. policy, runbook, api reference, release note, marketing page, ticket, transcript. if those live in one index with no type tag, similarity will happily mix them. humans do not. humans open the policy folder first. your retriever should be allowed the same prejudice.
version and product line matter as soon as you have more than one active release. this is not only for software. training manuals, insurance riders, and hr handbooks all have editions. if the filename encodes the version and you never parse it into a field, you are hoping the model notices a small header the chunker may have dropped. hope is not a filter.
locale and language are easy to ignore when the first corpus is english-only. the day you add a second language, unfiltered retrieval starts returning near-duplicates that look relevant and read wrong. tagging language at ingest is cheap. separating it later from mixed chunks is not.
effective dates and freshness are the quiet killers. "current" is not a property of text. it is a property of time relative to a document's validity window. without effective_from and effective_to (or a simple as_of convention), you will keep retrieving yesterday's truth with today's confidence. reranking by recency helps when newer is better. it does not help when the right answer is an older contract clause that is still binding.
access control metadata is not optional if more than one permission boundary exists in the corpus. even a coarse visibility field beats discovering, after a demo, that the index contains notes that were never meant for the role asking the question. retrieval that cannot respect tenancy is not a quality problem. it is a product problem wearing a relevance costume.
section path or heading breadcrumb deserves a mention because people confuse it with chunking. chunking decides what text unit you store. the path is metadata about where that unit lived. when two sections share similar sentences, the path is often the only cheap signal that one is "billing" and the other is "legal." store it. display it in citations. filter on it when the query implies a department.
store it even before you filter on it
teams delay metadata because they are not ready to expose filters in the ui. that is a product decision. it should not control what you persist.
if you store fields at ingest and ignore them at query time, you can turn them on later without reprocessing the world. if you skip the fields, every future filter request becomes a re-ingest project with archaeology attached: guess the version from a pdf footer, guess the audience from folder names, guess whether a doc is published from whether someone remembered to move it out of /drafts.
a practical rule: if a human would use a fact to decide whether a document is in scope, store that fact next to the chunk. you can start with five fields and a disciplined ingest path. filename conventions help. cms fields help more. "we will infer it with an llm later" is how you get confident wrong labels at scale.
also log the metadata on every retrieval trace. when an answer is wrong, you want to see not only the chunk text, but the labels that made it eligible. that is how you learn which fields actually move quality, instead of debating taxonomy in a vacuum.
how skipped metadata dresses up as a model problem
the ticket usually says the assistant is outdated, or confused, or "making things up." sometimes that is fair. more often the retrieved set included a document that should have been ineligible, and the model did what models do with locally plausible evidence.
watch for these tells. citations that bounce between product versions in one answer. answers that sound like landing pages when the question was operational. correct procedures pulled from a region the user is not in. internal shorthand leaking into external replies. high similarity scores on documents the team privately agrees are trash for this use case.
when you see those, resist the urge to start with prompt tone. print the candidate list with metadata columns. if the wrong row would have been removed by a one-line filter, your generation stack is not the bottleneck. your ingest contract is.
there is a mildly annoying truth here: good metadata makes demos less magical. you stop looking like a system that "just understands," and start looking like a system with rules. that is an improvement. users already live with rules when they use a normal search box. they do not need your retrieval layer to pretend the corpus is a single soup of equally trustworthy paragraphs.
if you are early, do not boil the ocean. pick the mismatches that already show up in manual support: wrong version, wrong audience, wrong freshness, wrong source type. encode those. keep the embedding model boring until the eligible set is sane. the cheapest retrieval win available to most teams is not a new index algorithm. it is refusing to retrieve documents you already knew were out of scope, back when you had the file in hand and chose to save only the text.