How to Build a Document Ingestion Pipeline for RAG in Python

How to Build a Document Ingestion Pipeline for RAG in Python

Table of Contents

What You Will Build

A reliable document ingestion pipeline for RAG is responsible for transforming raw documents into searchable knowledge. Before a retrieval-augmented generation system can answer questions, it must extract information from source files, organize that information into meaningful chunks, generate vector embeddings, and store the results in a searchable index.

In this tutorial, we will build a complete document ingestion pipeline for RAG using Python, Sentence Transformers, and FAISS. Rather than creating an isolated demonstration, we will extend the RAG application developed in our previous tutorials.

Our pipeline will accept PDF and TXT documents, extract their contents, normalize the text, split it into chunks, generate embeddings, and update a FAISS vector index. Each chunk will retain metadata that connects it to its original document.

Document Ingestion Architecture

The processing workflow will follow this sequence:

Document Upload
       |
       v
File Validation
       |
       v
Text Extraction
       |
       v
Text Cleaning
       |
       v
Document Chunking
       |
       v
Embedding Generation
       |
       v
FAISS Vector Index
       |
       v
Searchable RAG Knowledge Base

We will also implement document identifiers and update handling so that reprocessing a document does not silently create duplicate entries. This becomes particularly important when ingestion jobs are retried or documents are replaced with newer versions.

Integration with Our Existing RAG Application

In our previous tutorial, we introduced Celery and Redis to execute long-running document-processing operations outside the FastAPI request cycle. The background worker called a process_document() function, but its internal implementation was intentionally left for a separate tutorial.

We will now implement that function and connect it to the existing background task system. The resulting RAG ingestion pipeline will support asynchronous document processing without blocking interactive question-answering requests.

By the end of this guide, our document ingestion pipeline for RAG will provide the following capabilities:

  • Extract text from PDF and TXT documents.
  • Clean and normalize extracted content.
  • Split documents into searchable text chunks.
  • Generate embeddings using Sentence Transformers.
  • Store vectors and document metadata in FAISS.
  • Identify documents and safely handle updates.
  • Integrate ingestion with existing Celery background workers.
  • Make newly processed documents available for RAG retrieval.

The objective is not simply to create embeddings. We want a maintainable document-processing workflow that can be tested, extended, and eventually connected to a production document storage system.

Prerequisites

Before building our document ingestion pipeline for RAG, we need a Python environment and several libraries for document processing, embedding generation, and vector storage.

This tutorial continues the RAG application developed in our previous guides. We will reuse its FastAPI backend, Celery workers, and Redis configuration rather than rebuilding those components.

Required Technologies

  • Python 3.11+ — application and document-processing logic.
  • PyMuPDF — extracting text from PDF documents.
  • Sentence Transformers — generating dense vector embeddings.
  • FAISS — storing and searching document embeddings.
  • NumPy — preparing embedding arrays for FAISS.
  • FastAPI — accepting document uploads through the existing API.
  • Celery and Redis — executing ingestion jobs asynchronously.

Install the document-processing dependencies in the Python environment used by the application:

pip install pymupdf sentence-transformers faiss-cpu numpy

If FastAPI and Celery are already installed from our earlier tutorials, there is no need to reinstall them. However, the Celery worker environment must also contain the document-processing dependencies.

Project Structure

We will organize the document ingestion pipeline for RAG into separate modules so that extraction, chunking, embedding generation, and vector storage can be tested independently.

rag-project/
├── main.py
├── document_api.py
├── document_processor.py
├── rag_pipeline.py
├── ingestion/
│   ├── __init__.py
│   ├── extractor.py
│   ├── cleaner.py
│   ├── chunker.py
│   ├── embeddings.py
│   └── vector_store.py
├── tasks/
│   ├── __init__.py
│   ├── celery_app.py
│   └── document_tasks.py
├── data/
│   ├── uploads/
│   └── indexes/
├── requirements.txt
├── Dockerfile
└── docker-compose.yml

The ingestion/ package will contain reusable processing components. The document_processor.py module will coordinate these components, while rag_pipeline.py will preserve the interface expected by our existing RAG application.

The data/uploads/ directory will store incoming documents. The data/indexes/ directory will hold the FAISS index and its associated metadata.

For local development, these directories can exist on the same machine. In a multi-container deployment, the API and background workers must have access to the required shared storage. We will address index consistency and synchronization later in the tutorial.

Existing RAG Components

We assume that the application already has a working retrieval pipeline and an embedding model. The ingestion process must use the same embedding model and vector normalization strategy as the retrieval system.

Otherwise, the document vectors and query vectors may become incompatible, producing incorrect similarity scores or preventing FAISS searches entirely.

With these dependencies and interfaces established, we can begin designing the document-processing workflow.

Step 1 — Design the Document Ingestion Pipeline

Before implementing individual processing functions, we need to define how data moves through our document ingestion pipeline for RAG. A clear architecture makes it easier to test each component, identify processing failures, and extend the system without rewriting the entire pipeline.

Our design separates document processing into independent stages. Each stage receives data in a predictable format and passes its results to the next component.

Define the Processing Stages

The document ingestion pipeline for RAG will contain five main processing stages:

  • Extraction: read the document and extract its text.
  • Cleaning: normalize whitespace and remove unwanted characters.
  • Chunking: divide the text into smaller, meaningful segments.
  • Embedding: convert text chunks into numerical vectors.
  • Indexing: store vectors together with document metadata.

A coordinating function will execute these stages in the correct order. It will also validate intermediate results and report processing errors.

Define the Document Data Model

We need a consistent representation of the document and its metadata. For this tutorial, we will use Python dataclasses.

Create ingestion/models.py:

from dataclasses import dataclass


@dataclass
class DocumentChunk:
    document_id: str
    chunk_id: str
    text: str
    position: int


@dataclass
class ProcessingResult:
    document_id: str
    chunks_processed: int

Each DocumentChunk contains the text used to generate an embedding and the identifiers required to connect search results to their source document.

The document_id identifies the original document. The chunk_id identifies an individual chunk, while position preserves its order within the document.

We will generate deterministic chunk identifiers based on the document identifier and chunk position. This makes it possible to identify chunks consistently when a document is processed again.

Define the Pipeline Coordinator

The central coordinator will connect our processing components. For now, we will define its interface without implementing the individual stages.

Create document_processor.py:

from pathlib import Path

from ingestion.models import ProcessingResult


class TemporaryProcessingError(Exception):
    """A transient error that may succeed after a retry."""


def process_document(
    document_id: str,
    file_path: str
) -> dict:

    path = Path(file_path)

    if not path.is_file():
        raise FileNotFoundError(
            f"Document not found: {file_path}"
        )

    # The processing stages will be implemented
    # in the following sections:
    #
    # 1. Extract text
    # 2. Clean text
    # 3. Create chunks
    # 4. Generate embeddings
    # 5. Update the vector index

    raise NotImplementedError(
        "Document ingestion stages are not implemented yet."
    )

This is a temporary interface, not the finished implementation. We will replace the placeholder with working processing logic as we build the remaining components.

Notice that the function accepts the same arguments used by our existing Celery task: document_id and file_path. It will also return a dictionary containing the document identifier and the number of processed chunks.

This preserves compatibility with the background task system without requiring changes to the FastAPI endpoints.

Separate Permanent and Temporary Failures

Not every ingestion failure should trigger another attempt. An unsupported file format or an empty document is unlikely to become valid simply because the worker retries the task.

Temporary failures are different. A short network interruption or temporary storage unavailability may disappear after a delay.

Our TemporaryProcessingError exception provides a way to distinguish retryable failures from permanent ones. The Celery configuration created in the previous tutorial can retry tasks that raise this exception.

Individual processing components should raise ordinary exceptions for invalid input and use TemporaryProcessingError only when a retry has a reasonable chance of succeeding.

Keep Index Updates Consistent

One important requirement is that a failed ingestion job must not leave the knowledge base in a partially updated state.

For example, if embedding generation succeeds but index persistence fails, the system should not report the document as successfully processed.

We will therefore prepare the complete set of chunks and embeddings before updating the searchable index. Later, we will implement a replacement strategy that avoids accumulating duplicate vectors when documents are reprocessed.

This separation of responsibilities gives our RAG ingestion pipeline a predictable structure and prepares it for integration with Celery workers.

Step 2 — Extract Text from PDF and TXT Files

The first processing stage in our document ingestion pipeline for RAG is text extraction. Before we can generate embeddings, we need to convert documents into plain text that our chunking and embedding components can process.

We will support two common document formats: PDF and TXT. For PDF extraction, we will use PyMuPDF. Plain-text files will be read using Python’s built-in file handling.

Create the Text Extraction Module

Create ingestion/extractor.py:

from pathlib import Path

import pymupdf


SUPPORTED_EXTENSIONS = {".pdf", ".txt"}


def extract_pdf_text(file_path: Path) -> str:
    pages = []

    with pymupdf.open(file_path) as document:
        if document.needs_pass:
            raise ValueError(
                "Password-protected PDF files are not supported."
            )

        for page in document:
            text = page.get_text("text", sort=True)

            if text.strip():
                pages.append(text)

    return "\n\n".join(pages)


def extract_txt_text(file_path: Path) -> str:
    return file_path.read_text(encoding="utf-8-sig")


def extract_text(file_path: str | Path) -> str:
    path = Path(file_path)

    if not path.is_file():
        raise FileNotFoundError(
            f"Document not found: {path}"
        )

    extension = path.suffix.lower()

    if extension not in SUPPORTED_EXTENSIONS:
        raise ValueError(
            f"Unsupported document format: {extension}"
        )

    if extension == ".pdf":
        text = extract_pdf_text(path)
    else:
        text = extract_txt_text(path)

    if not text.strip():
        raise ValueError(
            "No extractable text found in the document."
        )

    return text

The main extract_text() function provides a single interface for both supported formats. It checks that the file exists, validates its extension, and selects the appropriate extraction method.

The function also rejects documents that contain no extractable text. This prevents empty documents from reaching later stages of our document ingestion pipeline for RAG.

Understand PDF Text Extraction

PDF documents are more complicated than ordinary text files. A PDF describes how content appears on a page, but its internal structure does not necessarily preserve the original reading order.

PyMuPDF extracts text from individual pages using page.get_text("text", sort=True). The sort=True option attempts to arrange extracted content in a more natural reading order based on its position.

However, this does not guarantee correct extraction from complex layouts containing multiple columns, tables, or floating text blocks.

Scanned PDFs present another limitation. If a document contains only images of text, ordinary PDF text extraction may return an empty result. Such documents require optical character recognition (OCR), which we will treat as a separate processing capability rather than silently applying it to every PDF.

For additional text extraction options, including layout handling and different output formats, see the official PyMuPDF text extraction documentation.

Handle Text File Encoding

For TXT documents, we use utf-8-sig encoding. This supports standard UTF-8 files and automatically handles a UTF-8 byte order mark when one is present.

Files using other encodings may raise UnicodeDecodeError. In a production ingestion system, we could add an explicit encoding-detection or conversion stage. For this tutorial, we require UTF-8 text files to keep the processing behavior predictable.

Test the Extraction Module

Create test_extraction.py in the project root:

from ingestion.extractor import extract_text


def main():
    file_path = "data/uploads/sample.pdf"

    text = extract_text(file_path)

    print("Extracted characters:", len(text))
    print()
    print(text[:1000])


if __name__ == "__main__":
    main()

Place a PDF containing selectable text in data/uploads/sample.pdf, then run:

python test_extraction.py

The script should display the number of extracted characters and the beginning of the document text.

Repeat the test with a UTF-8 TXT file to verify that both extraction methods work correctly.

Validate Extraction Before Continuing

Successful execution does not always mean the extracted text is suitable for retrieval. Inspect several documents and check whether paragraphs, headings, and important technical information remain understandable.

Pay particular attention to documents containing tables, multiple columns, headers, and footers. These elements can introduce noise or disrupt the logical order of extracted content.

At this stage, we are extracting text rather than attempting to reconstruct the complete visual structure of every document. The next processing stage will normalize the extracted content before chunking.

With a working extraction module, our RAG document processing pipeline can now accept PDF and TXT files through a consistent Python interface.

Step 3 — Clean and Normalize Extracted Text

Raw text extracted from documents is rarely ready for embedding generation. PDF files may contain broken line endings, repeated spaces, invisible characters, and formatting artifacts. These problems can reduce the quality of document chunks and make retrieved passages harder to understand.

In this step, we will add a text-cleaning module to our document ingestion pipeline for RAG. The objective is to remove unnecessary formatting while preserving the meaning and structure of the original document.

Why Text Cleaning Matters

Consider a paragraph extracted from a PDF:

Retrieval-augmented generation combines
information retrieval with
large language models.

The system retrieves    relevant
documents before generating an answer.

The line breaks and excessive spaces do not contribute meaningful information. They may also interfere with sentence-based chunking.

However, removing every line break would be a mistake. Paragraph boundaries, headings, and lists provide useful structural information that should remain available to the chunking algorithm.

A reliable document ingestion pipeline for RAG should therefore normalize formatting without converting the entire document into one continuous block of text.

Create the Text Cleaning Module

Create ingestion/cleaner.py:

import re
import unicodedata


def clean_text(text: str) -> str:
    if not text:
        return ""

    # Normalize Unicode representation.
    text = unicodedata.normalize("NFC", text)

    # Normalize line endings.
    text = text.replace("\r\n", "\n")
    text = text.replace("\r", "\n")

    # Replace non-breaking spaces.
    text = text.replace("\u00a0", " ")

    # Remove selected invisible formatting characters.
    text = re.sub(
        r"[\u200b\u200c\u200d\ufeff]",
        "",
        text
    )

    # Normalize horizontal whitespace.
    text = re.sub(r"[^\S\n]+", " ", text)

    # Remove spaces around line breaks.
    text = re.sub(r" *\n *", "\n", text)

    # Join lines inside paragraphs when the previous
    # line does not end with sentence punctuation.
    lines = text.split("\n")
    normalized_lines = []
    current = ""

    for line in lines:
        line = line.strip()

        if not line:
            if current:
                normalized_lines.append(current)
                current = ""

            normalized_lines.append("")
            continue

        if not current:
            current = line
        elif current.endswith((".", "!", "?", ":", ";")):
            normalized_lines.append(current)
            current = line
        else:
            current += " " + line

    if current:
        normalized_lines.append(current)

    text = "\n".join(normalized_lines)

    # Limit consecutive empty lines.
    text = re.sub(r"\n{3,}", "\n\n", text)

    return text.strip()

The function performs several normalization operations while preserving paragraph boundaries.

Unicode normalization ensures that equivalent character sequences use a consistent representation. Whitespace normalization removes unnecessary spacing, while the final processing stage attempts to reconstruct paragraphs broken across multiple PDF lines.

Test Text Normalization

Create a small test script:

from ingestion.cleaner import clean_text


sample_text = """
Retrieval-augmented generation combines
information retrieval with
large language models.

The system retrieves    relevant
documents before generating an answer.
"""

result = clean_text(sample_text)

print(result)

Expected output:

Retrieval-augmented generation combines information retrieval with large language models.

The system retrieves relevant documents before generating an answer.

This produces cleaner paragraphs while retaining the separation between independent blocks of information.

Avoid Aggressive Text Cleaning

Text normalization introduces an important trade-off. Removing formatting artifacts improves consistency, but excessive cleaning can destroy information that matters during retrieval.

For example, technical documents may contain:

  • Part numbers and equipment identifiers.
  • Mathematical expressions and measurement units.
  • Code examples and configuration parameters.
  • Numbered procedures and structured lists.
  • Tables containing related values.

These elements should not be modified simply to produce more uniform text.

Our line-joining heuristic is suitable for many ordinary prose documents, but it is not universally correct. In particular, it may combine list items, headings, or code lines that should remain separate.

For documents where layout is essential, a more advanced RAG ingestion pipeline should preserve structural elements during extraction instead of attempting to reconstruct them from plain text.

Connect Cleaning to Text Extraction

We can now combine the first two processing stages:

from ingestion.extractor import extract_text
from ingestion.cleaner import clean_text


def prepare_document_text(file_path: str) -> str:
    raw_text = extract_text(file_path)
    cleaned_text = clean_text(raw_text)

    if not cleaned_text:
        raise ValueError(
            "Document contains no usable text."
        )

    return cleaned_text

The resulting function extracts the original content, normalizes its formatting, and validates that usable text remains.

We will integrate these operations into process_document() after implementing the remaining processing stages.

At this point, our document ingestion pipeline for RAG can transform supported documents into normalized text. The next step is to divide that text into smaller segments suitable for embedding generation and semantic retrieval.

Step 4 — Split Documents into Chunks

After extracting and cleaning document text, the next stage in our document ingestion pipeline for RAG is chunking. Instead of generating one embedding for an entire document, we divide the content into smaller passages that can be retrieved independently.

Chunk size directly affects retrieval quality. Large chunks may contain too much unrelated information, while very small chunks can lose the context required to answer a question. Our goal is to preserve meaningful passages without exceeding the embedding model’s input limits.

Choose a Chunking Strategy

For this tutorial, we will implement a paragraph-aware chunker with overlapping text. It will attempt to keep paragraphs together, split oversized paragraphs into smaller pieces, and preserve context between adjacent chunks.

We will use two configuration parameters:

  • chunk_size: maximum number of characters in a chunk.
  • chunk_overlap: approximate number of characters repeated between adjacent chunks.

We will start with a chunk size of 800 characters and an overlap of 120 characters. These are initial testing values, not universal recommendations. The optimal settings depend on document structure, embedding model, and retrieval requirements.

Character-based limits are convenient for our initial implementation. However, production systems should also consider token counts because embedding models enforce token-based input limits.

Implement the Chunking Module

Create ingestion/chunker.py:

import re


def split_long_text(
    text: str,
    chunk_size: int
) -> list[str]:
    """Split oversized paragraphs at word boundaries."""

    words = text.split()
    chunks = []
    current = ""

    for word in words:
        if len(word) > chunk_size:
            if current:
                chunks.append(current)
                current = ""

            # Handle unusually long tokens.
            chunks.extend(
                word[i:i + chunk_size]
                for i in range(0, len(word), chunk_size)
            )
            continue

        candidate = f"{current} {word}".strip()

        if len(candidate) > chunk_size and current:
            chunks.append(current)
            current = word
        else:
            current = candidate

    if current:
        chunks.append(current)

    return chunks


def split_into_chunks(
    text: str,
    chunk_size: int = 800,
    chunk_overlap: int = 120
) -> list[str]:

    if chunk_size <= 0:
        raise ValueError("chunk_size must be positive")

    if not 0 <= chunk_overlap < chunk_size:
        raise ValueError(
            "chunk_overlap must be smaller than chunk_size"
        )

    if not text.strip():
        return []

    paragraphs = [
        paragraph.strip()
        for paragraph in re.split(r"\n\s*\n", text)
        if paragraph.strip()
    ]

    segments = []

    for paragraph in paragraphs:
        if len(paragraph) <= chunk_size:
            segments.append(paragraph)
        else:
            segments.extend(
                split_long_text(paragraph, chunk_size)
            )

    chunks = []
    current = ""

    for segment in segments:
        candidate = (
            f"{current}\n\n{segment}"
            if current else segment
        )

        if len(candidate) <= chunk_size:
            current = candidate
            continue

        if current:
            chunks.append(current)

        # Preserve context from the previous chunk.
        overlap = (
            current[-chunk_overlap:].lstrip()
            if chunk_overlap and current else ""
        )

        candidate = (
            f"{overlap}\n\n{segment}"
            if overlap else segment
        )

        # Avoid exceeding the configured size.
        if len(candidate) <= chunk_size:
            current = candidate
        else:
            current = segment

    if current:
        chunks.append(current)

    return chunks

The function first separates the document into paragraphs. Paragraphs that exceed the configured size are divided into smaller segments, preferably at word boundaries.

It then combines segments until adding another would exceed the maximum chunk size. When a new chunk begins, the function attempts to include a short overlap from the previous chunk.

This implementation keeps every chunk within the configured character limit. However, overlap is approximate: it may be shortened or omitted when the next segment already occupies most of the available space.

The overlap can also begin in the middle of a word or sentence. For more advanced applications, token-aware or sentence-aware overlap provides better control over chunk boundaries.

Create Document Chunk Objects

Our document ingestion pipeline for RAG must preserve the relationship between each chunk and its source document. We will use the DocumentChunk dataclass introduced earlier.

Add the following function to ingestion/chunker.py:

from ingestion.models import DocumentChunk


def create_document_chunks(
    document_id: str,
    text: str,
    chunk_size: int = 800,
    chunk_overlap: int = 120
) -> list[DocumentChunk]:

    text_chunks = split_into_chunks(
        text,
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap
    )

    return [
        DocumentChunk(
            document_id=document_id,
            chunk_id=f"{document_id}:{position}",
            text=chunk,
            position=position
        )
        for position, chunk in enumerate(text_chunks)
    ]

Each chunk receives a deterministic identifier consisting of the document ID and its position. Reprocessing the same document with the same chunking configuration produces the same identifiers.

These identifiers will help us replace document entries when updating the vector index. They do not guarantee that the chunk contents remain unchanged when the source document is modified.

Test the Chunking Strategy

Create test_chunking.py:

from ingestion.extractor import extract_text
from ingestion.cleaner import clean_text
from ingestion.chunker import create_document_chunks


raw_text = extract_text("data/uploads/sample.pdf")
cleaned_text = clean_text(raw_text)

chunks = create_document_chunks(
    document_id="sample-document",
    text=cleaned_text,
    chunk_size=800,
    chunk_overlap=120
)

print("Total chunks:", len(chunks))

for chunk in chunks[:3]:
    print()
    print("Chunk ID:", chunk.chunk_id)
    print("Length:", len(chunk.text))
    print("Text:", chunk.text[:200])

Run the test:

python test_chunking.py

Inspect the resulting chunks to verify that they contain meaningful passages rather than arbitrary fragments. Pay particular attention to paragraph boundaries, technical terms, and information that spans multiple sentences.

Evaluate Chunk Quality

Chunking should be evaluated as part of the complete retrieval pipeline, not only by inspecting character counts.

A useful experiment is to process the same document with several chunk sizes, such as 400, 800, and 1200 characters, and compare retrieval results using a fixed set of questions.

Smaller chunks may improve precision for narrowly focused questions. Larger chunks may preserve more context for complex explanations. Neither approach is consistently superior across all document types.

For a production RAG ingestion pipeline, we may eventually introduce different chunking strategies for technical manuals, source code, structured tables, and ordinary prose.

Our current implementation provides a reusable starting point. The next stage will convert these text chunks into numerical embeddings suitable for semantic search.

Step 5 — Generate Embeddings

After splitting documents into chunks, we need to convert their text into numerical vectors. This stage of our document ingestion pipeline for RAG allows the retrieval system to find passages based on semantic similarity rather than exact keyword matches.

We will use Sentence Transformers to generate dense embeddings. Each chunk will be represented by a fixed-length vector that can be stored in FAISS and compared with vectors generated from user queries.

Choose an Embedding Model

For this tutorial, we will use sentence-transformers/all-MiniLM-L6-v2. It is a relatively lightweight embedding model that produces 384-dimensional vectors and works well for demonstrating semantic retrieval in English-language documents.

The model is suitable for our initial implementation, but it has limitations. Its input length is restricted, and long passages may be truncated before embedding generation.

This is particularly important because our chunking stage currently uses character limits rather than token limits. An 800-character chunk is not guaranteed to fit within the embedding model’s maximum input length.

For multilingual documents, longer passages, or specialized technical content, a different embedding model may provide better results. Any model change must be coordinated with the existing retrieval system.

Create the Embedding Module

Create ingestion/embeddings.py:

from functools import lru_cache

import numpy as np
from sentence_transformers import SentenceTransformer


MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"


@lru_cache(maxsize=1)
def get_embedding_model() -> SentenceTransformer:
    return SentenceTransformer(MODEL_NAME)


def generate_embeddings(
    texts: list[str],
    batch_size: int = 32
) -> np.ndarray:

    if not texts:
        raise ValueError(
            "Cannot generate embeddings for an empty list."
        )

    if batch_size <= 0:
        raise ValueError(
            "batch_size must be positive."
        )

    if any(not text.strip() for text in texts):
        raise ValueError(
            "Embedding input contains empty text."
        )

    model = get_embedding_model()

    embeddings = model.encode(
        texts,
        batch_size=batch_size,
        convert_to_numpy=True,
        normalize_embeddings=True,
        show_progress_bar=False
    )

    return np.asarray(
        embeddings,
        dtype=np.float32
    )

The get_embedding_model() function loads the model only once per Python process. Subsequent calls reuse the same model instance instead of repeatedly loading its weights.

This is useful when our document ingestion pipeline for RAG runs inside a persistent Celery worker. The first task loads the model, while later tasks can reuse it.

The cache is process-local. If we run several Celery worker processes, each process may load its own model instance and consume additional memory.

Generate Embeddings in Batches

The batch_size parameter controls how many text chunks are processed together. Batch processing is generally more efficient than calling the embedding model separately for every chunk.

However, larger batches require more memory. The optimal batch size depends on the embedding model, available RAM or GPU memory, and the length of the input passages.

Our initial value of 32 provides a reasonable starting point. For large documents or memory-constrained workers, we can reduce it.

Normalize Embedding Vectors

We enable normalize_embeddings=True so that the generated vectors have approximately unit length.

This allows us to use inner-product similarity in FAISS as an equivalent ranking method for cosine similarity when both document and query embeddings are normalized.

The relationship is straightforward:

Normalized document vector + Normalized query vector
                         |
                         v
                 Inner Product
                         |
                         v
                Cosine Similarity

Normalization must be applied consistently. If document embeddings are normalized but query embeddings are not, the similarity scores will not have the expected interpretation.

Our existing RAG retrieval component must therefore use the same embedding model and normalization settings.

Test Embedding Generation

Create test_embeddings.py:

from ingestion.embeddings import generate_embeddings


texts = [
    "A RAG system retrieves relevant documents.",
    "FAISS performs similarity search over vectors.",
    "Document ingestion prepares knowledge for retrieval."
]

embeddings = generate_embeddings(texts)

print("Embedding shape:", embeddings.shape)
print("Data type:", embeddings.dtype)

print(
    "First vector norm:",
    (embeddings[0] ** 2).sum() ** 0.5
)

Run the script:

python test_embeddings.py

The expected embedding shape is:

Embedding shape: (3, 384)
Data type: float32
First vector norm: approximately 1.0

The first dimension represents the number of input passages. The second dimension represents the embedding model’s vector size.

The exact vector values will depend on the model, but their dimensions must remain consistent throughout the vector index.

Connect Embeddings to Document Chunks

We can now combine chunking and embedding generation:

from ingestion.chunker import create_document_chunks
from ingestion.embeddings import generate_embeddings


def embed_document(
    document_id: str,
    text: str
):
    chunks = create_document_chunks(
        document_id=document_id,
        text=text
    )

    if not chunks:
        raise ValueError(
            "Document produced no chunks."
        )

    embeddings = generate_embeddings(

[chunk.text for chunk in chunks]

) return chunks, embeddings

The function returns two related objects: a list of document chunks and a NumPy array containing their embeddings.

Their order must remain consistent. The vector at position i must correspond to the document chunk at the same position.

We will preserve this relationship when storing vectors and metadata in FAISS.

Prevent Silent Text Truncation

One important production consideration is the embedding model’s maximum input length.

Sentence Transformers may truncate text that exceeds the model’s configured token limit. This can silently discard information from the end of a chunk.

For our current model, we can inspect the configured limit:

from ingestion.embeddings import get_embedding_model


model = get_embedding_model()

print("Maximum sequence length:", model.max_seq_length)

Before processing production documents, we should measure the token lengths of our chunks and adjust the chunking configuration if necessary.

A more advanced RAG document processing pipeline would use the embedding model’s tokenizer to enforce token-aware chunk boundaries.

This is especially useful for documents containing code, technical terminology, or languages with different tokenization characteristics.

Prepare for Vector Storage

Our embedding module returns a two-dimensional NumPy array with the float32 data type expected by FAISS.

Before indexing, we will validate that the number of embeddings matches the number of chunks and that every vector has the expected dimensionality.

We will also preserve the model identifier alongside the index metadata. This helps prevent accidental mixing of embeddings generated by incompatible models.

At this stage, our document ingestion pipeline for RAG can extract documents, normalize their text, create chunks, and generate semantic embeddings.

The next step is to store those vectors in FAISS while preserving the metadata needed to identify and retrieve their source passages.

Step 6 — Store Embeddings and Metadata in FAISS

Our document ingestion pipeline for RAG can now transform documents into numerical embeddings. The next step is to store those vectors in a searchable index while preserving the information required to identify their original text.

We will use FAISS for vector similarity search and JSON for document metadata. This approach is suitable for a local RAG application and provides a foundation for the document update mechanism we will implement in the next step.

Understand Vector Storage and Metadata

FAISS stores numerical vectors and performs efficient similarity searches. However, it does not automatically maintain the original document text, chunk identifiers, or application-specific metadata.

We therefore need two related data structures:

  • FAISS index: stores embeddings for similarity search.
  • Metadata store: connects vector positions to document chunks.

Our storage architecture will look like this:

Document Chunks
       |
       v
Embedding Generation
       |
       v
+-------------------------+
| FAISS Vector Index      |
|                         |
| Vector 0                |
| Vector 1                |
| Vector 2                |
+-------------------------+
       |
       | Position mapping
       v
+-------------------------+
| Metadata Store          |
|                         |
| 0 → Document A, Chunk 0 |
| 1 → Document A, Chunk 1 |
| 2 → Document B, Chunk 0 |
+-------------------------+

The position of each metadata record must correspond to the position of its embedding in the FAISS index. Losing this relationship would cause the retrieval system to return incorrect source passages.

Create the FAISS Vector Store

Create ingestion/vector_store.py:

import json
import os
import tempfile
from pathlib import Path

import faiss
import numpy as np

from ingestion.embeddings import MODEL_NAME
from ingestion.models import DocumentChunk


class VectorStore:
    def __init__(
        self,
        storage_dir: str = "data/indexes"
    ):
        self.storage_dir = Path(storage_dir)
        self.storage_dir.mkdir(
            parents=True,
            exist_ok=True
        )

        self.index_path = (
            self.storage_dir / "documents.faiss"
        )

        self.metadata_path = (
            self.storage_dir / "metadata.json"
        )

        self.index = None
        self.metadata = []

    def create_index(self, dimension: int):
        if dimension <= 0:
            raise ValueError(
                "Vector dimension must be positive."
            )

        self.index = faiss.IndexFlatIP(dimension)
        self.metadata = []

    def add_chunks(
        self,
        chunks: list[DocumentChunk],
        embeddings: np.ndarray
    ):
        if not chunks:
            raise ValueError(
                "Cannot index an empty chunk list."
            )

        vectors = np.asarray(
            embeddings,
            dtype=np.float32
        )

        if vectors.ndim != 2:
            raise ValueError(
                "Embeddings must be a 2D array."
            )

        if len(chunks) != vectors.shape[0]:
            raise ValueError(
                "Chunk and embedding counts do not match."
            )

        if not np.isfinite(vectors).all():
            raise ValueError(
                "Embeddings contain invalid values."
            )

        if self.index is None:
            self.create_index(vectors.shape[1])

        if vectors.shape[1] != self.index.d:
            raise ValueError(
                "Embedding dimension mismatch."
            )

        records = [
            {
                "document_id": chunk.document_id,
                "chunk_id": chunk.chunk_id,
                "position": chunk.position,
                "text": chunk.text
            }
            for chunk in chunks
        ]

        self.index.add(
            np.ascontiguousarray(vectors)
        )

        self.metadata.extend(records)

    def save(self):
        if self.index is None:
            raise ValueError(
                "No FAISS index to save."
            )

        if self.index.ntotal != len(self.metadata):
            raise ValueError(
                "Index and metadata counts differ."
            )

        # Write temporary files before replacing
        # the current storage files.
        with tempfile.TemporaryDirectory(
            dir=self.storage_dir
        ) as temporary_dir:

            temp_dir = Path(temporary_dir)

            temp_index = temp_dir / "documents.faiss"
            temp_metadata = temp_dir / "metadata.json"

            faiss.write_index(
                self.index,
                str(temp_index)
            )

            metadata_document = {
                "embedding_model": MODEL_NAME,
                "dimension": self.index.d,
                "chunks": self.metadata
            }

            temp_metadata.write_text(
                json.dumps(
                    metadata_document,
                    ensure_ascii=False,
                    indent=2
                ),
                encoding="utf-8"
            )

            os.replace(
                temp_index,
                self.index_path
            )

            os.replace(
                temp_metadata,
                self.metadata_path
            )

    def load(self):
        if not self.index_path.exists():
            raise FileNotFoundError(
                "FAISS index not found."
            )

        if not self.metadata_path.exists():
            raise FileNotFoundError(
                "Metadata file not found."
            )

        metadata_document = json.loads(
            self.metadata_path.read_text(
                encoding="utf-8"
            )
        )

        if metadata_document["embedding_model"] != MODEL_NAME:
            raise ValueError(
                "Embedding model mismatch."
            )

        index = faiss.read_index(
            str(self.index_path)
        )

        if index.d != metadata_document["dimension"]:
            raise ValueError(
                "Stored vector dimension mismatch."
            )

        records = metadata_document["chunks"]

        if index.ntotal != len(records):
            raise ValueError(
                "Index and metadata are inconsistent."
            )

        self.index = index
        self.metadata = records

    def search(
        self,
        query_embedding: np.ndarray,
        top_k: int = 5
    ) -> list[dict]:

        if self.index is None:
            raise ValueError(
                "FAISS index is not loaded."
            )

        if top_k <= 0:
            raise ValueError(
                "top_k must be positive."
            )

        query = np.asarray(
            query_embedding,
            dtype=np.float32
        ).reshape(1, -1)

        if query.shape[1] != self.index.d:
            raise ValueError(
                "Query embedding dimension mismatch."
            )

        if not np.isfinite(query).all():
            raise ValueError(
                "Query embedding contains invalid values."
            )

        scores, positions = self.index.search(
            np.ascontiguousarray(query),
            top_k
        )

        results = []

        for score, position in zip(
            scores[0],
            positions[0]
        ):
            if position == -1:
                continue

            record = self.metadata[int(position)]

            results.append({
                **record,
                "score": float(score)
            })

        return results

We use faiss.IndexFlatIP, which performs exact inner-product similarity search. Because our embedding module generates normalized vectors, inner-product scores correspond to cosine similarity.

Unlike approximate nearest-neighbor indexes, IndexFlatIP does not require a training stage. This makes it straightforward to use for a relatively small document collection.

However, exact search compares the query against every stored vector. For very large collections, we may eventually need a more scalable FAISS index or a dedicated vector database.

Save the Index and Metadata

Our document ingestion pipeline for RAG must preserve processed documents between application restarts. The save() method writes two files:

  • documents.faiss — the serialized FAISS index.
  • metadata.json — chunk text, identifiers, and embedding model information.

We first write temporary files and then replace the existing files. This reduces the risk of leaving an individual file partially written.

There is an important limitation: replacing two files separately is not a transaction. If the process crashes between replacements, the FAISS index and metadata may belong to different versions.

For production deployments, a safer design uses immutable, versioned index directories and an atomic pointer to the active version. We will discuss this approach when implementing document updates and considering concurrent Celery workers.

Test Vector Storage

Create test_vector_store.py:

from ingestion.models import DocumentChunk
from ingestion.embeddings import generate_embeddings
from ingestion.vector_store import VectorStore


chunks = [
    DocumentChunk(
        document_id="doc-001",
        chunk_id="doc-001:0",
        text="FAISS provides vector similarity search.",
        position=0
    ),
    DocumentChunk(
        document_id="doc-001",
        chunk_id="doc-001:1",
        text="Celery executes background processing tasks.",
        position=1
    ),
    DocumentChunk(
        document_id="doc-002",
        chunk_id="doc-002:0",
        text="Embeddings represent text as numerical vectors.",
        position=0
    )
]

embeddings = generate_embeddings(

[chunk.text for chunk in chunks]

) store = VectorStore( storage_dir=”data/indexes/test” ) store.add_chunks( chunks, embeddings ) store.save() print(“Indexed vectors:”, store.index.ntotal) # Reload the saved index. loaded_store = VectorStore( storage_dir=”data/indexes/test” ) loaded_store.load() query = generate_embeddings([ “How can I search similar vectors?” ])[0] results = loaded_store.search( query_embedding=query, top_k=2 ) for result in results: print() print(“Document:”, result[“document_id”]) print(“Chunk:”, result[“chunk_id”]) print(“Score:”, round(result[“score”], 4)) print(“Text:”, result[“text”])

Run the test:

python test_vector_store.py

The script creates a FAISS index, stores three embeddings, saves the index, reloads it, and performs a semantic search.

The search results should contain relevant chunks together with their document identifiers, text, and similarity scores.

Understand Index Persistence Limitations

The current implementation provides the basic storage operations, but it does not yet support safe document replacement or concurrent index updates.

Calling add_chunks() repeatedly with the same document will append duplicate vectors. Additionally, two Celery workers must not independently load, modify, and overwrite the same index without coordination.

These problems cannot be solved simply by assigning deterministic chunk identifiers. FAISS does not automatically enforce uniqueness based on our metadata.

We will address document replacement and index consistency in the next step.

Another important consideration is index loading. FAISS index files should only be loaded from trusted storage because serialized index data is not a safe format for arbitrary untrusted input.

With vector storage implemented, our document ingestion pipeline for RAG now has the essential components required for semantic retrieval. The next task is to ensure that documents can be updated without creating duplicate vectors or corrupting the knowledge base.

Step 7 — Handle Duplicate Documents and Updates

A production-ready document ingestion pipeline for RAG must handle repeated uploads and document updates without accumulating duplicate vectors. Our current FAISS implementation supports adding chunks, but it does not automatically replace existing documents.

We will implement document replacement by rebuilding the index from the current metadata and the updated document. This approach is straightforward for smaller collections, although it becomes expensive as the knowledge base grows.

Implement Document Replacement

Add the following method to the VectorStore class in ingestion/vector_store.py:

from ingestion.embeddings import generate_embeddings


def replace_document(
    self,
    document_id: str,
    new_chunks: list[DocumentChunk]
):
    if not new_chunks:
        raise ValueError("Document contains no chunks.")

    if any(
        chunk.document_id != document_id
        for chunk in new_chunks
    ):
        raise ValueError("Document ID mismatch.")

    # Keep chunks belonging to other documents.
    remaining = [
        DocumentChunk(**record)
        for record in self.metadata
        if record["document_id"] != document_id
    ]

    all_chunks = remaining + new_chunks

    if not all_chunks:
        raise ValueError("No chunks available.")

    embeddings = generate_embeddings(

[chunk.text for chunk in all_chunks]

) # Build a replacement index in memory. new_index = faiss.IndexFlatIP( embeddings.shape[1] ) new_index.add( np.ascontiguousarray(embeddings) ) new_metadata = [ { “document_id”: chunk.document_id, “chunk_id”: chunk.chunk_id, “position”: chunk.position, “text”: chunk.text } for chunk in all_chunks ] self.index = new_index self.metadata = new_metadata

The method removes the previous chunks associated with the document and rebuilds the index using the remaining chunks and the updated content.

This provides logical replacement rather than appending duplicates. However, the operation regenerates embeddings for every retained chunk, so its computational cost increases with the collection size.

Make Updates Safe for Background Workers

The replacement method solves duplicate entries within one VectorStore instance, but concurrent workers introduce another problem.

If two Celery workers load the same index and independently save their changes, the second write can overwrite the first.

For our initial document ingestion pipeline for RAG, we will use a single dedicated index-writing worker and process index updates sequentially.

This requires more than setting worker concurrency to one: all index-writing tasks must be routed to the same queue, and only one worker instance may consume that queue.

The current two-file persistence method also needs improvement before production use. A safer approach is to save each complete index version in a separate directory and switch the active version only after both files have been written successfully.

Verify Document Replacement

After loading an existing index, call the new method with an updated document:

updated_chunks = [
    DocumentChunk(
        document_id="doc-001",
        chunk_id="doc-001:0",
        text="Updated information about FAISS indexing.",
        position=0
    )
]

store.replace_document(
    document_id="doc-001",
    new_chunks=updated_chunks
)

store.save()

print("Total vectors:", store.index.ntotal)

print(
    "Document chunks:",
    sum(
        record["document_id"] == "doc-001"
        for record in store.metadata
    )
)

The updated document should now have exactly one chunk, while chunks belonging to other documents remain available.

For larger collections, rebuilding the entire index is inefficient. A scalable alternative is to use a vector database that supports document-level upserts and deletions.

Our implementation establishes the replacement logic needed for a reliable RAG ingestion pipeline. In the next step, we will connect these components to the existing Celery task.

Step 8 — Connect the Pipeline to Celery

Our document ingestion pipeline for RAG now includes text extraction, cleaning, chunking, embedding generation, and vector storage. We can combine these components into a single function that Celery executes asynchronously.

Implement the Document Processor

Replace the placeholder in document_processor.py with the following implementation:

from pathlib import Path

from ingestion.extractor import extract_text
from ingestion.cleaner import clean_text
from ingestion.chunker import create_document_chunks
from ingestion.vector_store import VectorStore


class TemporaryProcessingError(Exception):
    """A transient error that may succeed after a retry."""


def process_document(
    document_id: str,
    file_path: str
) -> dict:

    path = Path(file_path)

    if not path.is_file():
        raise FileNotFoundError(file_path)

    text = clean_text(extract_text(path))

    if not text:
        raise ValueError("Document contains no usable text.")

    chunks = create_document_chunks(
        document_id=document_id,
        text=text
    )

    if not chunks:
        raise ValueError("Document produced no chunks.")

    store = VectorStore()

    if store.index_path.exists():
        store.load()

    store.replace_document(
        document_id=document_id,
        new_chunks=chunks
    )

    store.save()

    return {
        "document_id": document_id,
        "chunks_processed": len(chunks)
    }

The replace_document() method generates embeddings and rebuilds the FAISS index, so the coordinator does not need to repeat these operations.

Our existing Celery task can continue calling process_document() without changing its arguments or return format.

Configure a Dedicated Ingestion Queue

Route document-processing tasks to a dedicated queue by adding the following configuration to tasks/celery_app.py:

celery_app.conf.task_routes = {
    "rag.process_document": {
        "queue": "document_ingestion"
    }
}

Start a single worker for this queue:

celery -A tasks.celery_app:celery_app worker \
    --queues=document_ingestion \
    --concurrency=1 \
    --loglevel=INFO

Run only one consumer for this queue. This prevents concurrent index modifications by Celery workers, although it does not protect against writes from other application processes.

Make New Documents Searchable

After ingestion completes, the retrieval service must load the updated FAISS index. An API process holding an older index in memory will not automatically see the new vectors.

For local testing, explicitly reload the index after processing. In production, use versioned index snapshots and coordinate updates across API replicas.

The document ingestion pipeline for RAG is now connected to our asynchronous processing architecture. The next step is to verify that uploaded documents can be processed, indexed, and retrieved successfully.

Test the Document Ingestion Pipeline

Now that our document ingestion pipeline for RAG is connected to Celery, we need to verify that documents are processed correctly and become available for semantic search.

Upload a Test Document

Start FastAPI, Redis, and the dedicated Celery worker. Then upload a PDF document using the endpoint created in our previous tutorial:

curl -X POST http://localhost:8000/jobs/documents \
  -H "X-API-Key: YOUR_API_KEY" \
  -F "file=@sample.pdf"

The API should return HTTP 202 with a task identifier and document ID. Use the returned task ID to check processing status:

curl http://localhost:8000/jobs/TASK_ID \
  -H "X-API-Key: YOUR_API_KEY"

A successful task should return a result similar to:

{
  "task_id": "TASK_ID",
  "status": "SUCCESS",
  "result": {
    "document_id": "DOCUMENT_ID",
    "chunks_processed": 18
  }
}

Verify Semantic Retrieval

After processing completes, reload the FAISS index and search for information contained in the uploaded document.

from ingestion.embeddings import generate_embeddings
from ingestion.vector_store import VectorStore

store = VectorStore()
store.load()

query = generate_embeddings([
    "What information does the document contain?"
])[0]

results = store.search(query, top_k=3)

for result in results:
    print(result["document_id"])
    print(result["score"])
    print(result["text"][:200])

The returned passages should correspond to relevant sections of the uploaded document. Use specific questions about known document content to evaluate retrieval quality.

Test Document Updates and Failures

A reliable document ingestion pipeline for RAG must also behave predictably when documents are replaced or processing fails.

  • Process the same document ID twice and verify that chunk counts do not accumulate.
  • Replace a document with updated content and confirm that obsolete passages disappear.
  • Upload an empty or unsupported document and verify that the task fails appropriately.
  • Restart the application and confirm that the saved FAISS index remains readable.
  • Verify that the retrieval service reloads the updated index before searching.

For update testing, reuse the same document ID when calling process_document(). Uploading through the existing FastAPI endpoint generates a new ID and therefore creates a separate document.

These tests confirm the basic functionality of our RAG ingestion pipeline. They do not replace concurrency, recovery, and performance testing required before production deployment.

Production Considerations

Our document ingestion pipeline for RAG provides a working foundation for document processing. However, production deployment requires additional safeguards for storage consistency, performance, and fault tolerance.

Index Consistency and Concurrent Updates

The current implementation rebuilds the FAISS index when a document is updated. This is acceptable for small collections but becomes expensive as the number of documents increases.

A production system should use versioned index snapshots with atomic activation. This prevents readers from accessing mismatched index and metadata files.

A dedicated Celery worker reduces concurrent writes, but it does not provide distributed locking or transactional storage.

Embedding Performance and Resource Management

Embedding generation is one of the most resource-intensive stages of the document ingestion pipeline for RAG. Multiple workers may load separate model instances, increasing memory consumption.

For larger workloads, consider batching documents, separating embedding workers from indexing workers, and using GPU acceleration when justified by processing volume.

The current document replacement strategy also regenerates embeddings for unchanged chunks. Content hashes and incremental updates can eliminate unnecessary computations.

Document Validation and Security

Uploaded documents must be treated as untrusted input. Validate file size, content type, and supported formats before processing. Apply resource limits to prevent oversized documents from exhausting worker memory.

Store uploads outside publicly accessible directories and enforce authorization when retrieving or replacing documents.

Monitoring and Recovery

Track ingestion duration, processed chunks, embedding failures, queue length, and worker memory consumption.

Persist document-processing status separately from temporary Celery results. This allows the application to distinguish completed, failed, and interrupted ingestion jobs after restarts.

Finally, ensure that API replicas use the same active index version. A successful background task does not guarantee that every retrieval instance has loaded the updated knowledge base.

These improvements allow the RAG ingestion pipeline to evolve from a local implementation into a reliable distributed document-processing service.

Where to Go Next

We have built a document ingestion pipeline for RAG that extracts text, creates chunks, generates embeddings, and stores searchable vectors in FAISS. The pipeline also integrates with our existing Celery background workers.

To continue developing the RAG application, explore these related tutorials:

The next major improvement is incremental document indexing. Instead of rebuilding the entire vector index, an advanced document ingestion pipeline for RAG can detect modified content and update only the affected chunks.

Frequently Asked Questions

What is a document ingestion pipeline for RAG?

A document ingestion pipeline for RAG transforms raw documents into searchable information. It extracts text, cleans the content, splits it into chunks, generates embeddings, and stores vectors with their metadata. The retrieval system then uses these vectors to find relevant information.

Why do RAG systems split documents into chunks?

Chunking allows the retrieval system to find specific passages instead of returning entire documents. It also helps keep text within the embedding model’s input limits. Chunk size and overlap should be evaluated using representative documents and retrieval questions.

Can FAISS store document metadata?

FAISS primarily stores vectors for similarity search. Application metadata, including document identifiers and original text, must be maintained separately. Our implementation uses a JSON file that maps vector positions to document chunks.

How do you prevent duplicate documents in a RAG pipeline?

A document ingestion pipeline for RAG should assign stable document identifiers and replace existing entries when documents are updated. Our implementation rebuilds the index after removing the previous chunks associated with the same document ID.

Should document ingestion run in the background?

Background processing is recommended for operations that may take several seconds or minutes. Celery workers can process documents asynchronously while FastAPI continues handling interactive requests. This also enables task tracking, retries, and separate resource allocation.

How can a RAG ingestion pipeline handle millions of documents?

Large collections generally require incremental indexing, distributed processing, and scalable vector storage. Instead of rebuilding a FAISS index for every update, consider a vector database supporting efficient upserts and deletions, combined with persistent document metadata and coordinated background workers.

Conclusion

Building a reliable document ingestion pipeline for RAG is essential for transforming unstructured documents into searchable knowledge. In this tutorial, we implemented a Python workflow that extracts text from PDF and TXT files, cleans the content, creates document chunks, generates embeddings, and stores vectors in FAISS.

We also added document replacement logic and connected the pipeline to Celery for asynchronous processing. This allows our existing RAG application to process new documents without blocking interactive question-answering requests.

Our implementation provides a practical foundation, but larger deployments require incremental indexing, consistent index snapshots, and coordinated access across multiple workers.

With these components in place, our document ingestion pipeline for RAG can serve as the foundation for a more scalable, maintainable, and production-ready knowledge retrieval system.

Scroll to Top