Chunking and Metadata Strategy for Enterprise RAG

Chunking strategy decides whether a RAG system over contracts and policy documents gives you a defensible answer or a hallucination dressed up in confident prose, more than retrieval quality does. I've watched three separate enterprise deployments stall for months because someone assumed the default tutorial settings would hold up on legal text. They didn't, and the postmortems all pointed to the same root cause: the chunker.

Why fixed-length chunking breaks contracts

Split a wiki page or a blog post into 512-token windows with some overlap and you'll get reasonable results. That's the default in every RAG tutorial you'll find, and it's fine for the use case it was built for. Contracts are a different animal.

A termination clause in a master services agreement almost always cross-references a definitions section three pages earlier and a liability cap two pages later. Chop the agreement into fixed windows and you'll end up with a chunk that says "either party may terminate for cause as defined in Section 2.3," except Section 2.3 lives in a different chunk, retrieved or not depending on how the vector math shakes out for that particular query. The model then fabricates a definition, or worse, states the wrong one with total confidence. I've seen this exact failure surface a defined term from an entirely unrelated exhibit, and the output read as authoritative as anything else the system produced. Legal teams do not forgive that. A wrong citation in a contract review carries real exposure.

Policy documents fail the same way, just with a different shape. HR handbooks and compliance manuals are hierarchical on purpose: a policy statement, then exceptions, then exceptions to the exceptions, then a regional addendum that quietly overrides everything above it. A fixed chunk boundary doesn't know any of that exists. It will cut a sentence in half just as readily as it will sever an "employees must" clause from the "except in California" carve-out sitting one paragraph down.

Structure-aware chunking as the baseline

The fix that's held up across the implementations I've seen work is structure-aware chunking: split along the document's own boundaries, its sections, numbered clauses, defined terms, rather than an arbitrary token count. Contracts practically hand you this for free. Most are built from templates, with numbered articles, lettered subsections, and defined terms flagged in caps or bold. Pull that structure out, whether through regex on the numbering scheme or a layout-aware PDF parser that keeps heading levels intact, and your chunk boundaries start respecting the document's logic instead of cutting against it.

This matters more for contracts than for almost any other document type companies feed into RAG, because contract language assumes the reader has the whole instrument in view. A confidentiality clause that says "as set forth above" means nothing on its own. Chunk by clause or section, tag each chunk to its parent section, and attach the defined-terms glossary as metadata, and the referential structure survives the process of tearing the document apart for embedding.

There's a second layer worth building on top of this: parent-child chunking. Embed and retrieve against a small chunk, a single clause, but hand the model the larger parent section as context once that clause is retrieved. Embedding an entire twenty-page section produces a vector so diluted it matches everything and, functionally, nothing; a single clause gives you a sharp, specific vector. But the generation step still needs the surrounding text to interpret that clause correctly, so you retrieve small and generate large. Anthropic published guidance on this in 2024 under the name "contextual retrieval": prepend a short, model-generated summary of the surrounding document to each chunk before embedding, and retrieval accuracy improves measurably, because you're putting back the context the raw chunk lost when it got sliced out.

Metadata is the filter that makes retrieval usable

Chunking gets the content right. Metadata makes that content findable, and trustworthy, at the scale an enterprise actually operates at. It's also the part every proof-of-concept skips, because it doesn't show up well in a demo and nobody wants to build a permissions layer for a pilot.

Try answering this with semantic similarity alone: "What's our indemnification exposure across all vendor agreements signed after 2022 in the EMEA region?" The word "indemnification" might not even appear in the clause you actually need. No embedding model infers "after 2022" or "EMEA" from vector distance, full stop. That query needs structured filters sitting on top of the semantic search: effective date, governing jurisdiction, counterparty, contract type, department owner, expiration date, amendment status. Every chunk has to inherit this metadata from its parent document so the retrieval step can narrow the candidate pool before similarity search runs at all.

Versioning is worth calling out on its own, because it's the failure mode that does the most damage once a system is live. Contracts get amended. Policies get revised. If your vector store is holding three versions of the same NDA template with no effective-date or supersession field attached, the retriever has no way to prefer the current one over the dead one, and sooner or later it will surface language that stopped being valid two years ago. You need an explicit "superseded_by" or "status: active/expired" field, checked at query time, not after the fact. This isn't a novel problem. SharePoint and Iron Mountain have treated version control as a first-class concern for decades. Bolting a RAG pipeline onto a document store without inheriting that discipline means giving up a capability the older system already provided.

Access permissions belong in the same bucket: unglamorous, and completely non-negotiable. A retriever that ignores document-level permissions and surfaces a chunk from an executive severance agreement to whoever happens to ping the internal Slack bot has built a data leak. Row-level security needs enforcement at the retrieval layer, filtering candidate chunks by the requesting user's actual permissions before anything reaches the model, not as a courtesy check on the answer afterward.

Provenance and citation: the difference between a tool and a liability

Ask any buyer evaluating RAG for legal or compliance work what they actually care about, and it comes down to one question: can I act on this without going back to re-read the source myself? The honest answer is only if the system can show its work, down to the paragraph.

That means every chunk needs enough metadata to rebuild an exact citation: document title, section number, page number, ideally a stable clause or paragraph identifier, something more useful than "contractv3final.pdf" as a label. Harvey and Hebbia, two vendors that built their reputations specifically in legal and financial services, converged on this because their buyers won't accept prose they can't verify against the source document in under thirty seconds. Lawyers don't work on faith. A RAG system that hands back confident, well-formed prose with no traceable citation is, to that audience, indistinguishable from a guess that happens to be dressed well.

Chunk size is a tuning problem, not a settled question

There is no universal right chunk size for contracts, and I'd be skeptical of any vendor who claims otherwise. Smaller chunks, a single clause or paragraph, often somewhere in the 100 to 300 token range, improve precision because the embedding represents one coherent idea instead of a blend of three or four. Larger chunks hold onto more context but dilute the vector, and a diluted vector struggles to tell a highly relevant clause apart from one that's merely sitting nearby in the document.

Most mature implementations land on a hybrid: chunk small for embedding and retrieval, expand to the parent section for generation, the pattern described above. Then layer keyword or BM25 search on top of the vector search, because legal text is dense with exact strings, defined terms, statute citations, specific dollar figures, that lexical matching catches and semantic search sometimes lets slip past. Hybrid search, sparse plus dense, has become close to standard practice in serious enterprise builds for exactly this reason.

Maintenance is the part nobody budgets for

None of this is a one-time build, and treating it like one is how these systems quietly rot. Contracts get amended, policies get revised, regulations shift underneath documents that were compliant the day they were signed. A chunking and metadata pipeline that isn't wired into the document lifecycle, triggering re-chunking and re-embedding on every material edit, updating supersession fields automatically, will drift out of sync with reality within a few months.

That drift doesn't announce itself. It sits there, invisible, until someone acts on stale information and the cost shows up as a bad decision made with total, false confidence. That's the exact failure enterprise RAG was supposed to prevent, and reproducing it in a new form is a costly outcome.