What It Actually Takes to Build a RAG Pipeline Over Real Contracts

Enterprises don't have a data problem. They have an unstructured data problem. Petabytes of contracts, policies, engineering drawings, and compliance documents sit in file shares and object stores — readable by humans, invisible to search. The value isn't missing. It's locked behind the fact that nobody built a way in.

I wanted to see exactly what it takes to build that "way in," end to end, using real documents instead of a toy dataset. This is a walkthrough of the pipeline: the architecture decisions, the chunking strategy, and — because production ML infrastructure never behaves quite like the tutorial — the environment realities that show up the moment you leave a single well-behaved notebook.

The dataset: real contracts, not synthetic text

I used the Contract Understanding Atticus Dataset (CUAD) — 510 real commercial legal contracts, professionally annotated across 41 clause categories (governing law, indemnification, IP assignment, non-compete, and so on). It's a good stand-in for the kind of document enterprises actually have too much of: long, inconsistently formatted, legally dense, and mostly unsearchable beyond Ctrl+F.

That inconsistency turned out to matter more than any other single decision in the pipeline.

Architecture

Raw contracts → Object storage (S3-compatible)
      ↓
Structure-aware chunking (section-boundary + recursive token windows)
      ↓
Embedding (BGE-M3, batch-encoded on GPU)
      ↓
Vector index (Qdrant, with clause-type + source metadata)
      ↓
Retrieval (top-k dense search)
      ↓
Generation (Qwen3-14B, served via vLLM)
      ↓
Grounded, cited answer

Every stage of this mirrors a real enterprise pattern: unstructured files at rest in object storage, a retrieval layer that turns that content into something structured and queryable, and a generation layer that answers in natural language while staying anchored to the actual source documents.

Chunking: the part everyone underestimates

Naive fixed-size chunking — split every 500 characters, done — breaks contracts mid-clause. A 500-character window has no idea it just cut a sentence in half between "the Company shall indemnify" and the actual scope of that indemnification three lines later.

Contracts have real structure: Articles, Sections, numbered sub-clauses. So the chunking pipeline runs two passes:

1. Structure-aware split — find section/clause boundaries (ARTICLE IV, Section 8.1, numbered headers) and split there first, so each chunk starts at a natural boundary instead of an arbitrary character offset.
2. Recursive windowing — for any section still too long, split further into ~450-word windows with roughly 15% overlap, so clauses that reference earlier definitions ("as defined in Section 3.2") don't lose that context entirely.

Across 510 contracts this produced just over 19,000 chunks — a real illustration of the fan-out that happens once you move from "one document" to "a real corpus."

Worth being honest about: regex-based section detection gets you most of the way, not all of the way. Contract formatting in the wild is inconsistent enough that some documents fall back to whole-section chunks rather than clean clause-level splits. That's not a flaw unique to this pipeline — it's the actual shape of the problem enterprises face when they say their data is "unstructured."

Embedding and retrieval

Chunks are embedded with BGE-M3, a strong, efficiently sized open embedding model, and indexed into Qdrant alongside metadata — source filename, chunk position, and CUAD's own clause-type labels where available. That metadata layer is what turns "semantic search" into something closer to structured retrieval: filtering to a clause type or a specific contract before the vector search even runs, rather than relying on the embedding alone to do all the work.

Generation: serving an open model with vLLM

Answers are generated by Qwen3-14B, served through vLLM on a single GPU:

vllm serve Qwen/Qwen3-14B \
  --served-model-name qwen3-14b \
  --dtype bfloat16 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.80 \
  --host 0.0.0.0 --port 8000

The retrieval step feeds the top-k matching chunks into a prompt that explicitly instructs the model to answer only from the provided excerpts, cite the source contract and section for every claim, and say so plainly when the retrieved context doesn't contain enough information — rather than filling the gap with a plausible-sounding guess. That instruction is doing real work: it's the difference between a system that summarizes documents and one that hallucinates convincingly, and it's the single most important design choice for anything touching legal or compliance content.

The part the tutorials skip: environment reality

Every RAG tutorial assumes a GPU environment that already works. Real ones don't, and the gap between pip install vllm and a running server is where most of the actual engineering time goes. A few things worth knowing if you're doing this yourself:

None of this is a knock on the tools. It's the actual texture of standing up inference infrastructure, and it's exactly the kind of operational reality that separates "I ran a notebook once" from being able to speak credibly to how this works in production.

Closing the loop: what a real query looks like

Once everything is wired together, a question like "What are common termination-for-convenience provisions across these contracts?" triggers the full pipeline: the question is embedded, the top-k relevant clauses are retrieved from Qdrant across the 510-contract corpus, and the model generates an answer that cites the specific contract and section each claim came from — rather than a generic, ungrounded summary.

That's the actual deliverable here, and the actual pitch underneath it: enterprises don't need a chatbot that sounds confident. They need a system that can find the right needle in an enormous, messy, unstructured haystack — and show its work when it does.

This pipeline was built end-to-end on a single rented GPU instance, using open-source models and tools throughout: CUAD as the source corpus, S3-compatible object storage, Qdrant for vector search, BGE-M3 for embeddings, and Qwen3-14B served via vLLM for generation.