The Enterprise RAG Pipeline: 7 Stages From Ingestion to a Cited Answer
A RAG pipeline turns an organization's documents into grounded answers: it ingests content, parses and chunks it, enriches and indexes it, retrieves the passages relevant to a question, and has a model answer from them with citations. An enterprise RAG pipeline also carries source permissions, document versions and an evaluation loop through every stage.
Key takeaways
- A RAG pipeline has seven stages: connect and ingest, parse and chunk, enrich, index, retrieve, generate, and evaluate and observe.
- In enterprises, most failures happen before the model: lost permissions, scanned PDFs, superseded versions and duplicate copies.
- Permissions must be captured at ingestion and enforced inside the retrieval query, never filtered after the model has read the text.
- Good retrieval is hybrid: vector search for meaning, keyword search for exact identifiers, and graph context for relationships.
- Every answer should cite its passages and be checked against them, and the pipeline should be scored continuously on real questions.
A RAG pipeline is the chain of stages that turns an organization's documents into grounded answers: it ingests content, parses and indexes it, retrieves the passages relevant to a question, and has a language model answer from those passages with citations. An enterprise RAG pipeline adds what a demo skips: source permissions on every chunk, current document versions, hybrid retrieval and a loop that scores answers on real questions.
The demo version takes an afternoon. Point an embedding model at a folder, store vectors, retrieve the top five chunks, ask a model to answer. It works on twenty clean PDFs. It fails on 35,000 operational documents with scans, superseded revisions, folders only three people may open, and part numbers that embeddings round off. The difference is not the model. It is the six stages around it.
What is a RAG pipeline?
Retrieval-augmented generation was described by Lewis et al. in 2020 as combining a language model with a retriever over an external document index, so the model generates from retrieved evidence rather than only from what it learned in training. The enterprise version keeps that idea and wraps it in the controls a company needs: who may read what, which version is current, and how to prove where an answer came from.
The comparison with fine-tuning is covered in RAG vs. fine-tuning. For most knowledge work, retrieval wins because documents change weekly and permissions differ by person, and neither fits inside model weights.
The 7 stages of an enterprise RAG pipeline at a glance
| Stage | What it does | What breaks in enterprises | The fix |
|---|---|---|---|
| 1. Connect and ingest | Pulls content from source systems | Permissions lost in the copy; deletions never reach the index | Capture each source's access list with every record; fail closed; reconcile deletions |
| 2. Parse and chunk | Turns files into clean passages | Scans read as blank; tables flattened; clauses cut in half | OCR and table extraction; chunk on document structure |
| 3. Enrich | Adds metadata, versions and entities | Superseded revisions answer as current; duplicates compete | Version and owner metadata; entity extraction; one canonical copy |
| 4. Index | Makes passages searchable | Embeddings miss exact codes; mixed embedding models | Hybrid index (vector + keyword + graph); record the embedding model per chunk |
| 5. Retrieve | Finds the passages for a question | Restricted text reaches the model; relevant passage ranked 40th | Permission filter inside the query; rank fusion and reranking |
| 6. Generate | Writes the answer from passages | Unsupported claims; citations that don't back the sentence | Citation-first answers; check against sources; say "not covered" |
| 7. Evaluate and observe | Measures and records quality | "It seemed good" with no evidence; silent regressions | Scored question sets; per-answer source record; release gates |
Stage 1: Connect and ingest with permissions
Ingestion decides most of what can go wrong later. A connector's real job is not to copy documents. It is to carry each document's access-control list across intact and keep group memberships current, so a permission check at question time still means something.
Three defaults matter. A source with no permission rows is closed, not public. A failed membership lookup restricts the source rather than opening it. And a document deleted at the source must stop answering, through a change feed or a periodic full sweep. The connector side is covered in detail in permission-preserving RAG connectors.
In SphereIQ Connect, a webhook or schedule fires, the event lands in a durable change-data-capture queue, bursts are coalesced into one incremental delta sync, and content is chunked and embedded with secrets redacted at ingest and source permissions captured with every record.
Stage 2: Parse and chunk
Enterprise documents are rarely clean text. They are scanned forms, contracts with numbered clauses, runbooks with steps, spreadsheets exported to PDF, and drawings with tables of part numbers.
- Parse what is actually there. Scans need OCR. Tables need table extraction, or a rate table becomes a jumble of numbers with no headers.
- Chunk along structure. Split on headings, clauses, procedure steps and table rows rather than fixed character windows. A chunk that ends mid-clause answers half a question.
- Keep context on the chunk. The document title, section heading and identity travel with each passage, so it still makes sense when retrieved alone.
Document intelligence (parsing, OCR and table extraction) is what took a routine document question at a professional-services firm from six hours to minutes.
Stage 3: Enrich with metadata, versions and entities
Enrichment is where an index becomes a knowledge layer. Each chunk should carry the source system, owner, last-modified date and version, and the entities it mentions: products, sites, customers, policies.
Versions are the most common silent failure. If version 3 and version 4 of the travel policy are both indexed, retrieval will happily cite whichever scores higher. Answers in a regulated plant must be cited to the controlled revision, not to a copy someone saved to a shared drive. Fix it at the source where possible, and mark superseded content so it stops answering.
Entities and shared definitions matter too. "The Atlanta plant" and "Site 4" should resolve to one node, and "net revenue" should mean one formula. That is the job of a knowledge graph and a semantic layer.
Stage 4: Index with vectors, keywords and a graph
A good enterprise index is hybrid:
- Vector index for meaning: the passage that says the same thing in different words.
- Keyword index (for example BM25) for exact strings: error codes, part numbers, clause references, names.
- Graph for relationships: who owns this system, which procedure references that part.
One rule is easy to miss. Embeddings from different models are not comparable, so each chunk should record the model that embedded it, and each search should run against one model at a time.
Stage 5: Retrieve with access control
Retrieval must apply permissions inside the query, so a passage the person asking cannot read is never a candidate. It is not reranked, not read by the model and not placed in the context window. Filtering after the model has read the text is a promise, not a wall. The reasoning is laid out in permission-aware RAG.
In the Company Brain, semantic search over embeddings and lexical keyword search run together, results are fused by reciprocal-rank fusion, an optional reranker reorders for relevance, and the knowledge graph expands to connected context. Every step is filtered by the asker's source permissions.
Most RAG failures happen before the model runs: a permission dropped at ingestion, a scan read as blank, an old version that outranks the current one. Fix the first five stages and the model has far less to get wrong.
Stage 6: Generate with grounding and citations
Generation should be citation-first. Every claim in the answer points to the passage that supports it, and when the retrieved passages do not cover the question, the system says so instead of filling the gap.
Two practical points. Context order matters: research on long contexts found that models use information at the start or end of a long input more reliably than information in the middle, so fewer, better-ranked passages usually beat a stuffed context window. And answers should be checked against their sources before delivery. In SphereIQ, answers return inline [n] citations, groundedness is scored against the cited passages, and low-confidence outputs are flagged, blocked or routed to a person according to policy.
Stage 7: Evaluate and observe
A pipeline without measurement drifts. Score retrieval and answers separately:
- Retrieval. Did the right passage come back? Citation hit-rate against ground-truth answers is the cleanest measure.
- Answers. Reviewers score correctness, completeness, sourcing, honesty and usefulness; the method is in how to score a RAG answer.
- Gating. Changes to chunking, models or prompts must pass the same question set before release.
- Records. Every answer should be logged with the passages it drew on, so a low score can be traced to its cause. What to capture is covered in LLM observability.
Enterprise RAG architecture: where the pipeline runs
The seven stages describe what the pipeline does. Architecture decides where each stage runs and what crosses your boundary, and that is usually the first question a security review asks.
Three things leave a typical RAG deployment: document text sent for embedding, passages sent to the model for generation, and optional calls such as reranking or OCR services. If any one of them skips your approved route, the data-residency claim is false, however the diagram looks. A self-hosted pipeline is only as private as its model calls, and the useful test is a network whose only exit is your own AI gateway. That test is described in self-hosted LLMs in your VPC.
SphereIQ runs in three ways: Cloud, hosted by Sphere; Dedicated, a single-tenant environment in your own AWS account managed by Sphere; and Private, in your own cloud, where you hold the key that seals stored credentials. In every model, model calls go only to the providers you enable, optional features such as reranking stay off until you turn them on, and SphereIQ does not train models on your data.
The other architectural choice is ownership. Teams that build their own pipeline own all seven stages, including connector permission handling and deletion reconciliation, which is where most of the long-term effort goes. Teams that run a platform own the content, the definitions and the test questions. Either way, the five tests below apply.
How to test an enterprise RAG pipeline before you trust it
Run five checks on your own content, not a vendor's sample:
- Ask a question as a user who lacks access to the source that holds the answer. The right result is nothing.
- Delete a document at the source, wait one sync cycle, and ask about it again.
- Ask about a part number or clause reference that appears in exactly one document.
- Ask a question your documents do not answer. The system should say so.
- Index two versions of one policy and check which one gets cited.
If the pipeline passes all five, the remaining work is content: owners, versions and definitions. The checklist in AI-ready data covers that side.
Frequently asked questions
What is a RAG pipeline?
What makes an enterprise RAG pipeline different from a basic one?
What is the best chunking strategy for RAG?
Should a RAG pipeline use vector search or keyword search?
How do you evaluate a RAG pipeline?
Run the pipeline on your own documents.
In a walkthrough we connect one of your sources, including a restricted one, and trace a real question through every stage to a cited answer.