How to Build Incremental Indexing for RAG in Python

How to Build Incremental Indexing for RAG in Python

Table of Contents

What You Will Build

A traditional document ingestion pipeline processes documents, generates embeddings, and stores vectors in a searchable index. However, when documents are updated, rebuilding the entire index can waste processing time and computational resources.

In this tutorial, we will implement incremental indexing for RAG using Python, Sentence Transformers, and FAISS. Instead of reprocessing the complete knowledge base, our system will identify changed content and update only the affected vectors and metadata.

We will extend the document ingestion pipeline developed in our previous tutorial. The existing application already supports PDF and TXT extraction, text cleaning, chunking, embedding generation, FAISS storage, and asynchronous processing with Celery and Redis.

Incremental Indexing Architecture

Our new indexing workflow will compare incoming documents against previously processed versions before deciding which operations are necessary.

Document Upload
       |
       v
Calculate Document Hash
       |
       v
Compare with Stored Version
       |
       v
   Content Changed?
       |
       +---- No ----> Skip Processing
       |
      Yes
       |
       v
Extract and Chunk Document
       |
       v
Compare Chunk Hashes
       |
       v
+---------------------------+
| Unchanged Chunks          | --> Keep Existing Vectors
| New or Modified Chunks    | --> Generate Embeddings
| Deleted Chunks            | --> Remove Old Vectors
+---------------------------+
       |
       v
Update FAISS Index
       |
       v
Synchronize Metadata
       |
       v
Updated RAG Knowledge Base

The main objective of incremental indexing for RAG is to avoid unnecessary embedding generation while keeping the vector index synchronized with the latest document content.

What We Will Implement

Our implementation will introduce several capabilities:

  • Document hashing: detect whether a document has changed since its previous ingestion.
  • Chunk-level comparison: identify unchanged, new, modified, and removed content.
  • Embedding reuse: preserve existing vectors for unchanged chunks.
  • Selective indexing: insert new vectors and remove obsolete entries from FAISS.
  • Metadata synchronization: maintain consistent relationships between documents, chunks, and vector identifiers.
  • Background processing: integrate incremental updates with our existing Celery workers.

Extending the Existing RAG Application

In the previous tutorial, our VectorStore used faiss.IndexFlatIP and maintained chunk metadata in a separate JSON file. Updating a document required regenerating embeddings and rebuilding the complete index.

We will replace that strategy with an index that supports explicit vector identifiers. This allows us to remove outdated vectors and add new ones without recomputing embeddings for the entire collection.

We will also introduce a persistent chunk registry containing content hashes and vector IDs. This registry will help us determine which embeddings can be reused when a document changes.

An important distinction is that incremental indexing does not automatically make updates transactional. FAISS and our metadata store must still be coordinated carefully to prevent inconsistent states after interrupted processing.

We will address these limitations as part of the implementation rather than assuming that individual vector operations guarantee a reliable production system.

Expected Result

By the end of this tutorial, our RAG application will be able to process an updated document while preserving embeddings for unchanged chunks. Repeated ingestion of identical content will not generate duplicate vectors or unnecessary embedding requests.

The resulting incremental indexing for RAG workflow will provide a more efficient foundation for continuously updated knowledge bases, technical documentation, and document question-answering applications.

Prerequisites

Before implementing incremental indexing for RAG, we need to prepare the existing Python application and establish a consistent approach to vector identifiers, document metadata, and persistent storage.

This tutorial continues our previous document ingestion project. We will reuse its text extraction, cleaning, chunking, embedding generation, and Celery background-processing components.

Required Technologies

  • Python 3.11+ — application logic and document processing.
  • FAISS — vector similarity search with explicit vector IDs.
  • Sentence Transformers — generating embeddings for new and modified chunks.
  • NumPy — preparing vector arrays for FAISS.
  • SQLite — persistent document and chunk metadata.
  • Celery and Redis — asynchronous document-processing tasks.

The required libraries should already be installed from our previous tutorials. If necessary, install the core dependencies:

pip install faiss-cpu sentence-transformers numpy celery redis

SQLite and the hashlib module are included in Python’s standard library, so they require no additional installation.

Update the Project Structure

We will extend the existing RAG indexing pipeline with modules responsible for change detection, persistent metadata, and selective vector updates.

rag-project/
├── main.py
├── document_api.py
├── document_processor.py
├── ingestion/
│   ├── __init__.py
│   ├── extractor.py
│   ├── cleaner.py
│   ├── chunker.py
│   ├── embeddings.py
│   ├── vector_store.py
│   ├── hashing.py
│   ├── registry.py
│   └── incremental_indexer.py
├── tasks/
│   ├── __init__.py
│   ├── celery_app.py
│   └── document_tasks.py
├── data/
│   ├── uploads/
│   └── indexes/
├── requirements.txt
└── docker-compose.yml

The new modules will have separate responsibilities:

  • hashing.py — calculate document and chunk hashes.
  • registry.py — track document versions, chunk records, and vector IDs.
  • incremental_indexer.py — coordinate change detection and index updates.
  • vector_store.py — extend FAISS operations to support explicit vector identifiers.

Preserve Embedding Compatibility

Our incremental indexing for RAG implementation will continue using the embedding model configured in the previous tutorial:

MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"

This model produces 384-dimensional embeddings. Both document and query vectors must use the same model and normalization settings.

Changing the embedding model requires rebuilding the affected vector collection. Even when two models produce vectors with identical dimensions, their embedding spaces are not necessarily compatible.

Prepare Persistent Storage

We will store document and chunk metadata in SQLite, while FAISS will continue handling vector search.

Our storage layout will be:

data/
├── uploads/
└── indexes/
    ├── vectors.faiss
    └── registry.db

SQLite will track document hashes, chunk hashes, and stable vector IDs. FAISS will store the corresponding embeddings.

These are separate storage systems, so updating SQLite and FAISS cannot be treated as a single database transaction. We will introduce coordinated persistence and recovery safeguards later in the tutorial.

For the initial implementation, we will use one dedicated indexing worker. Multiple workers writing to the same local FAISS index would require additional synchronization.

With these prerequisites established, we can begin implementing incremental indexing for RAG by defining how document changes are detected and translated into vector operations.

Step 1 — Understand Incremental Indexing

Traditional document indexing often processes an entire collection whenever its content changes. This approach is simple to implement, but it becomes increasingly expensive as the number of documents and embeddings grows.

Incremental indexing for RAG solves this problem by identifying changes and updating only the affected parts of the knowledge base. Unchanged content retains its existing embeddings, while new or modified content is processed separately.

Full Reindexing vs. Incremental Indexing

Consider a knowledge base containing 10,000 document chunks. If a document update changes only five chunks, full reindexing may regenerate all 10,000 embeddings.

With incremental indexing, we aim to generate embeddings only for the five changed chunks, provided that their boundaries and identifiers can be matched reliably.

OperationFull ReindexingIncremental Indexing
Unchanged chunksReprocessedExisting embeddings reused
New chunksProcessedProcessed
Modified chunksProcessedUpdated selectively
Deleted chunksRemoved during rebuildRemoved by vector ID
Index update costDepends on entire collectionDepends mainly on changes

Understand the Three Levels of Change Detection

Our incremental indexing for RAG workflow will distinguish three levels of change.

  1. Document level: compare document hashes to determine whether the source content has changed.
  2. Chunk level: compare individual chunk hashes to identify content that can reuse existing embeddings.
  3. Vector level: add new embeddings and remove obsolete vectors using explicit FAISS identifiers.

Document hashes provide a fast way to skip unchanged files. Chunk hashes provide finer control when only part of a document has been modified.

Why Chunk Positions Are Not Reliable Identifiers

In our previous implementation, chunk identifiers were based on document IDs and positions:

document-001:0
document-001:1
document-001:2

This works for basic indexing, but inserting a paragraph near the beginning of a document can shift the positions of later chunks.

Consequently, chunk position alone cannot determine whether the underlying content has changed.

We will introduce content hashes and persistent vector identifiers. Identical chunk content can then be recognized even when its position changes.

Repeated passages require special handling because two chunks may have the same content hash. Our registry will therefore track individual chunk occurrences rather than assuming that every content hash is unique.

Choose a FAISS Index That Supports Vector IDs

Our previous implementation used faiss.IndexFlatIP, where vectors were referenced by their positions in the index.

For selective updates, we will use faiss.IndexIDMap2 around an inner-product index:

import faiss

dimension = 384

index = faiss.IndexIDMap2(
    faiss.IndexFlatIP(dimension)
)

This configuration supports explicitly assigned integer vector IDs through add_with_ids() and removal through remove_ids().

The underlying IndexFlatIP still performs exact similarity search. Incremental updates avoid rebuilding the entire index, but search complexity remains proportional to the number of stored vectors.

Plan for Consistent Updates

A successful RAG incremental indexing operation must keep three elements synchronized: document metadata, chunk records, and FAISS vectors.

We will use SQLite to track the metadata and FAISS to store embeddings. Because these systems do not share a transaction mechanism, we must design a recovery strategy for interrupted updates.

Our initial implementation will use a single indexing worker, stable vector IDs, and coordinated persistence. Later sections will address recovery and consistency between the registry and the saved index.

With this architecture established, the next step is to implement document and chunk hashing, which will form the basis of our incremental indexing for RAG workflow.

Step 2 — Track Documents with Content Hashes

The first practical component of incremental indexing for RAG is content hashing. A cryptographic hash allows us to determine whether a document has changed without comparing its entire contents against a previously processed version.

We will use SHA-256 from Python’s standard library. The same hashing mechanism will also help us identify unchanged text chunks and reuse their existing embeddings.

Create the Hashing Module

Create ingestion/hashing.py:

import hashlib
from pathlib import Path


def calculate_file_hash(
    file_path: str | Path
) -> str:
    """Calculate SHA-256 from the original file bytes."""

    path = Path(file_path)
    hasher = hashlib.sha256()

    with path.open("rb") as file:
        for block in iter(
            lambda: file.read(1024 * 1024),
            b""
        ):
            hasher.update(block)

    return hasher.hexdigest()


def calculate_chunk_hash(text: str) -> str:
    """Calculate SHA-256 for normalized chunk text."""

    return hashlib.sha256(
        text.encode("utf-8")
    ).hexdigest()

The calculate_file_hash() function reads documents in 1 MB blocks. This avoids loading an entire large PDF into memory just to calculate its hash.

The calculate_chunk_hash() function hashes UTF-8 text. We will apply it after extraction, cleaning, and chunking so that the hash represents the exact text sent to the embedding model.

Test Document Change Detection

Create test_hashing.py:

from ingestion.hashing import (
    calculate_file_hash,
    calculate_chunk_hash
)


file_hash = calculate_file_hash(
    "data/uploads/sample.pdf"
)

chunk_a = "FAISS supports vector similarity search."
chunk_b = "FAISS supports vector similarity search."
chunk_c = "FAISS supports efficient vector search."

print("Document hash:", file_hash)

print(
    "Identical chunks:",
    calculate_chunk_hash(chunk_a)
    == calculate_chunk_hash(chunk_b)
)

print(
    "Modified chunks:",
    calculate_chunk_hash(chunk_a)
    == calculate_chunk_hash(chunk_c)
)

The first comparison should return True, while the second should return False. Even a small change in the text produces a different SHA-256 hash.

Distinguish File Changes from Content Changes

An important detail in incremental indexing for RAG is that file hashes and chunk hashes answer different questions.

A PDF can have a different binary hash after its metadata changes, even when the extracted text remains identical. Conversely, a small text modification can change the document hash while affecting only one or two chunks.

Our workflow will therefore use two checks:

  • Document hash: skip extraction when the file is identical to its previously processed version.
  • Chunk hash: reuse existing embeddings when the extracted content contains unchanged chunks.

Chunk hashes must be interpreted together with the embedding model and preprocessing configuration. If we change the embedding model, text-cleaning rules, or chunking strategy, previously generated embeddings may no longer be compatible with the new pipeline.

We will account for these configuration changes when deciding whether stored embeddings can be reused.

Handle Repeated Chunk Content

Two different sections of a document may contain identical text. Both sections will produce the same content hash, but they still represent separate occurrences within the document.

For example:

Document A
    Chunk 0 → Hash ABC
    Chunk 1 → Hash DEF
    Chunk 2 → Hash ABC

The first and third chunks have identical content but different positions. We must not assume that their metadata records or vector identifiers are interchangeable.

Our SQLite registry will track each occurrence separately. When matching updated chunks, we will reuse existing records where possible and assign new identifiers only when necessary.

This provides the foundation for selective embedding reuse without losing the relationship between retrieved passages and their original documents.

What Hashing Does Not Solve

Content hashing detects changes, but it does not automatically update FAISS or synchronize metadata.

It also does not prevent chunk boundaries from shifting when paragraphs are inserted or removed. Such changes may cause additional chunks to receive new hashes even when much of their underlying text remains similar.

Nevertheless, hashing gives our incremental indexing for RAG pipeline a reliable mechanism for identifying identical content and avoiding unnecessary processing.

In the next step, we will create a persistent document registry that stores these hashes and identifies new, modified, unchanged, and deleted documents.

Step 3 — Detect New, Modified, and Deleted Documents

To implement incremental indexing for RAG, we need a persistent registry that remembers which documents have already been processed. File hashes alone are insufficient unless we can compare them with previously stored values.

We will use SQLite to track document identifiers, content hashes, and processing status. The registry will distinguish new, modified, and unchanged documents. Deleted documents will be identified through an explicit deletion operation or a source synchronization process.

Create the Document Registry

Create ingestion/registry.py:

import sqlite3
from pathlib import Path


class DocumentRegistry:
    def __init__(
        self,
        db_path: str = "data/indexes/registry.db"
    ):
        self.db_path = Path(db_path)
        self.db_path.parent.mkdir(
            parents=True,
            exist_ok=True
        )

        self.initialize()

    def connect(self):
        return sqlite3.connect(self.db_path)

    def initialize(self):
        with self.connect() as connection:
            connection.execute("""
                CREATE TABLE IF NOT EXISTS documents (
                    document_id TEXT PRIMARY KEY,
                    file_hash TEXT NOT NULL,
                    status TEXT NOT NULL,
                    updated_at TEXT DEFAULT CURRENT_TIMESTAMP
                )
            """)

            connection.execute("""
                CREATE TABLE IF NOT EXISTS chunks (
                    vector_id INTEGER PRIMARY KEY,
                    document_id TEXT NOT NULL,
                    chunk_hash TEXT NOT NULL,
                    position INTEGER NOT NULL,
                    text TEXT NOT NULL,
                    FOREIGN KEY(document_id)
                        REFERENCES documents(document_id)
                )
            """)

            connection.execute("""
                CREATE INDEX IF NOT EXISTS
                idx_chunks_document
                ON chunks(document_id)
            """)

    def get_document(self, document_id: str):
        with self.connect() as connection:
            connection.row_factory = sqlite3.Row

            row = connection.execute(
                """
                SELECT document_id, file_hash, status
                FROM documents
                WHERE document_id = ?
                """,
                (document_id,)
            ).fetchone()

            return dict(row) if row else None

    def detect_change(
        self,
        document_id: str,
        file_hash: str
    ) -> str:

        document = self.get_document(document_id)

        if document is None:
            return "NEW"

        if (
            document["file_hash"] == file_hash
            and document["status"] == "READY"
        ):
            return "UNCHANGED"

        return "MODIFIED"

    def get_chunks(self, document_id: str):
        with self.connect() as connection:
            connection.row_factory = sqlite3.Row

            rows = connection.execute(
                """
                SELECT vector_id, chunk_hash, position, text
                FROM chunks
                WHERE document_id = ?
                ORDER BY position
                """,
                (document_id,)
            ).fetchall()

            return [dict(row) for row in rows]

The documents table stores the last successfully indexed version of each document. The chunks table stores individual chunk occurrences, their hashes, and stable FAISS vector IDs.

We will add transactional update methods to this registry when implementing vector synchronization.

Detect Document Changes

The detect_change() method returns one of three states:

  • NEW: the document identifier does not exist in the registry.
  • UNCHANGED: the stored hash matches and the document is ready.
  • MODIFIED: the hash differs or the stored document is not ready.

For example:

from ingestion.hashing import calculate_file_hash
from ingestion.registry import DocumentRegistry


registry = DocumentRegistry()

document_id = "manual-001"

file_hash = calculate_file_hash(
    "data/uploads/manual.pdf"
)

state = registry.detect_change(
    document_id,
    file_hash
)

print("Document state:", state)

For a document that has never been indexed, the result will be NEW. After successful ingestion and registry synchronization, an identical upload should return UNCHANGED.

Detect Deleted Documents

Deletion requires a different approach. An uploaded document cannot be considered deleted simply because it was absent from the latest upload request.

For a synchronized document collection, we can compare the set of currently available document IDs with the IDs stored in SQLite.

Add this method to DocumentRegistry:

def find_deleted_documents(
    self,
    active_document_ids: set[str]
) -> list[str]:

    with self.connect() as connection:
        rows = connection.execute(
            "SELECT document_id FROM documents"
        ).fetchall()

    stored_ids = {row[0] for row in rows}

    return sorted(
        stored_ids - active_document_ids
    )

This method is appropriate only when active_document_ids represents a complete, authoritative inventory of the managed collection. It must not be used with a partial upload batch.

The returned identifiers are deletion candidates. Their vectors and metadata will be removed together during a coordinated index update.

Keep Registry Updates Consistent

A key requirement of incremental indexing for RAG is that SQLite must not report a document as successfully indexed before the corresponding FAISS changes are durable.

Our registry currently provides read operations and change detection. Later, we will implement coordinated writes, stable vector ID allocation, and recovery handling.

For now, the important result is that the application can identify which documents require processing without generating embeddings or modifying the vector index.

Step 4 — Identify Changed Chunks

Document-level hashing tells us whether a file has changed, but it does not identify which passages require new embeddings. For efficient incremental indexing for RAG, we need to compare the chunks of an updated document with those stored in our SQLite registry.

Our comparison algorithm will preserve existing vector IDs for unchanged chunks, identify new content requiring embeddings, and mark obsolete vectors for removal.

Match Chunks by Content Hash

Chunk positions can change when paragraphs are inserted or removed. Therefore, matching chunks only by their position would cause unnecessary embedding generation.

Instead, we will group existing chunks by their content hashes. Each stored occurrence remains a separate record, allowing identical passages to appear multiple times in the same document.

Create ingestion/chunk_diff.py:

from collections import defaultdict, deque
from dataclasses import dataclass

from ingestion.hashing import calculate_chunk_hash
from ingestion.models import DocumentChunk


@dataclass
class ChunkDiff:
    unchanged: list[tuple[int, DocumentChunk]]
    added: list[DocumentChunk]
    removed_ids: list[int]


def compare_chunks(
    old_records: list[dict],
    new_chunks: list[DocumentChunk]
) -> ChunkDiff:

    available = defaultdict(deque)

    for record in old_records:
        available[record["chunk_hash"]].append(
            record
        )

    unchanged = []
    added = []
    reused_ids = set()

    for chunk in new_chunks:
        chunk_hash = calculate_chunk_hash(
            chunk.text
        )

        candidates = available[chunk_hash]

        # Verify actual text to avoid relying
        # exclusively on hash equality.
        match = None

        for record in list(candidates):
            if record["text"] == chunk.text:
                match = record
                candidates.remove(record)
                break

        if match is None:
            added.append(chunk)
        else:
            vector_id = match["vector_id"]

            unchanged.append(
                (vector_id, chunk)
            )

            reused_ids.add(vector_id)

    removed_ids = [
        record["vector_id"]
        for record in old_records
        if record["vector_id"] not in reused_ids
    ]

    return ChunkDiff(
        unchanged=unchanged,
        added=added,
        removed_ids=removed_ids
    )

The function produces three groups:

  • Unchanged: chunks that can retain their existing embeddings and vector IDs.
  • Added: chunks that require embedding generation and new vector IDs.
  • Removed: old vector IDs that no longer belong to the updated document.

A modified chunk is represented as one removed occurrence and one added occurrence. This simplifies the vector update logic because FAISS can remove the obsolete vector and insert the replacement.

Test Chunk Comparison

Consider a document containing three chunks. After editing, the second chunk changes while the first and third remain identical.

from ingestion.chunk_diff import compare_chunks
from ingestion.hashing import calculate_chunk_hash
from ingestion.models import DocumentChunk


old_texts = [
    "Introduction to RAG",
    "Original embedding explanation",
    "FAISS vector search"
]

old_records = [
    {
        "vector_id": i + 100,
        "chunk_hash": calculate_chunk_hash(text),
        "position": i,
        "text": text
    }
    for i, text in enumerate(old_texts)
]

new_texts = [
    "Introduction to RAG",
    "Updated embedding explanation",
    "FAISS vector search"
]

new_chunks = [
    DocumentChunk(
        document_id="doc-001",
        chunk_id=f"doc-001:{i}",
        text=text,
        position=i
    )
    for i, text in enumerate(new_texts)
]

diff = compare_chunks(
    old_records,
    new_chunks
)

print("Unchanged:", len(diff.unchanged))
print("Added:", len(diff.added))
print("Removed:", len(diff.removed_ids))

Expected output:

Unchanged: 2
Added: 1
Removed: 1

Only the updated passage requires a new embedding. The other two chunks retain their existing vectors.

Handle Repeated Content and Configuration Changes

The comparison also handles repeated text. If a document contains three identical chunks and the updated version contains only two, the algorithm reuses two existing vector IDs and removes the third.

This approach preserves the correct number of chunk occurrences. However, content hashes alone do not establish semantic identity between different occurrences of identical text.

Our incremental indexing for RAG implementation also assumes that the existing vectors were generated using the same embedding model and compatible preprocessing settings.

If the embedding model changes, the application must invalidate the previous embeddings rather than reuse them based solely on matching text hashes.

Prepare the Vector Update

The resulting ChunkDiff object contains everything needed for selective indexing:

Unchanged chunks → Preserve vector IDs
Added chunks     → Generate new embeddings
Removed IDs      → Delete obsolete FAISS vectors

Metadata positions must also be updated for reused chunks because their locations within the document may have changed.

With chunk comparison implemented, our incremental indexing for RAG pipeline can determine which vectors to preserve, add, or remove without regenerating embeddings for unchanged content.

Step 5 — Update FAISS Without Rebuilding the Entire Index

Now that we can identify changed chunks, the next step is to update the vector index selectively. Our previous implementation used IndexFlatIP, which assigned vector positions automatically.

For incremental indexing for RAG, we will use IndexIDMap2 to associate each embedding with a persistent 64-bit integer identifier.

Create an Index with Explicit Vector IDs

We will update ingestion/vector_store.py to support insertion, deletion, searching, and persistence.

from pathlib import Path

import faiss
import numpy as np


class VectorStore:
    def __init__(
        self,
        storage_dir="data/indexes",
        dimension=384
    ):
        self.storage_dir = Path(storage_dir)
        self.storage_dir.mkdir(
            parents=True,
            exist_ok=True
        )

        self.index_path = (
            self.storage_dir / "vectors.faiss"
        )

        self.dimension = dimension

        if self.index_path.exists():
            self.index = faiss.read_index(
                str(self.index_path)
            )

            if self.index.d != dimension:
                raise ValueError(
                    "FAISS dimension mismatch"
                )

            if not isinstance(
                self.index,
                faiss.IndexIDMap2
            ):
                raise ValueError(
                    "Expected an IndexIDMap2 index"
                )
        else:
            self.index = faiss.IndexIDMap2(
                faiss.IndexFlatIP(dimension)
            )

    def add_vectors(
        self,
        embeddings,
        vector_ids
    ):
        vectors = np.asarray(
            embeddings,
            dtype=np.float32
        )

        ids = np.asarray(
            vector_ids,
            dtype=np.int64
        )

        if vectors.ndim != 2:
            raise ValueError(
                "Embeddings must be a 2D array"
            )

        if vectors.shape != (
            len(ids),
            self.dimension
        ):
            raise ValueError(
                "Embedding dimensions do not match"
            )

        if len(set(ids.tolist())) != len(ids):
            raise ValueError(
                "Duplicate vector IDs in batch"
            )

        if len(ids) == 0:
            return

        self.index.add_with_ids(vectors, ids)

    def remove_vectors(self, vector_ids):
        ids = np.asarray(
            vector_ids,
            dtype=np.int64
        )

        if len(ids) == 0:
            return 0

        return self.index.remove_ids(ids)

    def search(self, query_embedding, top_k=5):
        query = np.asarray(
            query_embedding,
            dtype=np.float32
        ).reshape(1, -1)

        scores, ids = self.index.search(
            query,
            top_k
        )

        return [
            {
                "vector_id": int(vector_id),
                "score": float(score)
            }
            for vector_id, score
            in zip(ids[0], scores[0])
            if vector_id != -1
        ]

    def save(self):
        temporary_path = self.index_path.with_suffix(
            ".faiss.tmp"
        )

        faiss.write_index(
            self.index,
            str(temporary_path)
        )

        temporary_path.replace(
            self.index_path
        )

Unlike the previous vector store, this implementation does not maintain a parallel JSON metadata list. Document and chunk metadata will be retrieved from our SQLite registry using the returned vector IDs.

Apply Selective Vector Updates

The ChunkDiff object created in Step 4 identifies which vectors must be removed and which chunks require new embeddings.

The following example demonstrates the FAISS operations. It assumes that unique vector IDs have already been allocated by the metadata registry.

from ingestion.embeddings import generate_embeddings


def apply_vector_changes(
    store,
    diff,
    new_vector_ids
):
    if len(new_vector_ids) != len(diff.added):
        raise ValueError(
            "Vector ID count does not match new chunks"
        )

    if diff.added:
        embeddings = generate_embeddings(

[chunk.text for chunk in diff.added]

) else: embeddings = None store.remove_vectors( diff.removed_ids ) if diff.added: store.add_vectors( embeddings, new_vector_ids ) return { “reused”: len(diff.unchanged), “added”: len(diff.added), “removed”: len(diff.removed_ids) }

This function modifies the in-memory FAISS index. It deliberately does not save the index or update SQLite.

We will coordinate those operations in the next step so that metadata is not committed independently of vector changes.

Verify the Index Behavior

A simple test can confirm that explicit vector identifiers remain stable after deletion and insertion.

import numpy as np

from ingestion.vector_store import VectorStore


store = VectorStore(
    storage_dir="data/test_index"
)

rng = np.random.default_rng(42)

vectors = rng.random(
    (3, 384)
).astype(np.float32)

vectors /= np.linalg.norm(
    vectors,
    axis=1,
    keepdims=True
)

store.add_vectors(
    vectors,
    [101, 102, 103]
)

store.remove_vectors([102])

print("Vector count:", store.index.ntotal)

The expected vector count is 2. The remaining vectors retain IDs 101 and 103.

The test assumes an empty test index. Run it in a fresh directory to avoid inserting identifiers that already exist.

Important FAISS Limitations

IndexIDMap2 supports explicit identifiers, but it does not automatically prevent duplicate IDs. Our registry must allocate unique IDs and ensure that existing IDs are never inserted a second time.

The underlying IndexFlatIP stores vectors in memory and performs exact similarity search. Removing and adding vectors avoids regenerating the entire embedding collection, but these operations are not equivalent to a database transaction.

For additional implementation details, see the official FAISS index documentation.

We now have the essential vector operations for incremental indexing for RAG. The next step is to assign persistent IDs, synchronize SQLite records, and ensure that interrupted updates can be recovered safely.

Step 6 — Synchronize Metadata and Vector IDs

Our FAISS index now supports selective vector updates, but incremental indexing for RAG requires more than adding and removing embeddings. Every vector ID must correspond to the correct document, chunk text, and position in SQLite.

Because SQLite and FAISS cannot participate in one shared transaction, we will use versioned FAISS snapshots. The registry will record which snapshot represents the committed index.

Add Index Version Tracking

Extend the initialize() method in ingestion/registry.py with a table for the active index generation:

connection.execute("""
    CREATE TABLE IF NOT EXISTS index_state (
        id INTEGER PRIMARY KEY CHECK (id = 1),
        generation INTEGER NOT NULL
    )
""")

connection.execute("""
    INSERT OR IGNORE INTO index_state
    (id, generation)
    VALUES (1, 0)
""")

Generation zero represents an empty index. Each successful update creates a new FAISS snapshot and advances the generation stored in SQLite.

Allocate Persistent Vector IDs

Vector IDs must remain unique across updates. Rather than deriving IDs from chunk positions or hashes, we will allocate them from a persistent SQLite sequence.

Add the following table to initialize():

connection.execute("""
    CREATE TABLE IF NOT EXISTS vector_id_sequence (
        id INTEGER PRIMARY KEY AUTOINCREMENT
    )
""")

Then add this method to DocumentRegistry:

def allocate_vector_ids(self, count: int) -> list[int]:
    if count < 0:
        raise ValueError("Count cannot be negative")

    ids = []

    with self.connect() as connection:
        for _ in range(count):
            cursor = connection.execute(
                "INSERT INTO vector_id_sequence DEFAULT VALUES"
            )
            ids.append(cursor.lastrowid)

    return ids

Allocated IDs are never reused, even if an indexing attempt fails. This is acceptable: vector IDs identify records and do not need to form a continuous sequence.

Save Versioned FAISS Snapshots

Update the persistence logic in VectorStore. Instead of overwriting the active index file, save each committed generation under a separate filename.

def save_generation(self, generation: int):
    final_path = (
        self.storage_dir
        / f"vectors-{generation}.faiss"
    )

    temporary_path = (
        self.storage_dir
        / f"vectors-{generation}.faiss.tmp"
    )

    if final_path.exists():
        raise FileExistsError(final_path)

    faiss.write_index(
        self.index,
        str(temporary_path)
    )

    temporary_path.replace(final_path)

    return final_path

The new snapshot is written before SQLite references it. A crash during snapshot creation therefore leaves the previous committed generation available.

For production durability, the temporary file and containing directory also require appropriate filesystem synchronization. The example above provides a practical starting point, not a complete crash-durability guarantee.

Commit Document and Chunk Metadata

We will update document metadata, chunk records, and the active generation in one SQLite transaction.

Add this method to DocumentRegistry:

def commit_document_update(
    self,
    document_id: str,
    file_hash: str,
    chunk_records: list[dict],
    expected_generation: int,
    new_generation: int
):
    with self.connect() as connection:
        connection.execute("BEGIN IMMEDIATE")

        current = connection.execute(
            "SELECT generation FROM index_state WHERE id = 1"
        ).fetchone()[0]

        if current != expected_generation:
            raise RuntimeError(
                "Index generation changed during update"
            )

        connection.execute(
            """
            INSERT INTO documents
                (document_id, file_hash, status, updated_at)
            VALUES (?, ?, 'READY', CURRENT_TIMESTAMP)
            ON CONFLICT(document_id)
            DO UPDATE SET
                file_hash = excluded.file_hash,
                status = 'READY',
                updated_at = CURRENT_TIMESTAMP
            """,
            (document_id, file_hash)
        )

        connection.execute(
            "DELETE FROM chunks WHERE document_id = ?",
            (document_id,)
        )

        connection.executemany(
            """
            INSERT INTO chunks
                (vector_id, document_id,
                 chunk_hash, position, text)
            VALUES (?, ?, ?, ?, ?)
            """,
            [
                (
                    record["vector_id"],
                    document_id,
                    record["chunk_hash"],
                    record["position"],
                    record["text"]
                )
                for record in chunk_records
            ]
        )

        connection.execute(
            """
            UPDATE index_state
            SET generation = ?
            WHERE id = 1
            """,
            (new_generation,)
        )

This transaction ensures that SQLite does not expose a partially updated set of chunk records.

The chunk_records list must include both reused chunks and newly embedded chunks, with their updated positions.

Coordinate the Update Sequence

Our incremental indexing for RAG workflow will follow this order:

1. Read the committed index generation
2. Load its FAISS snapshot
3. Compare old and new chunks
4. Allocate IDs for new chunks
5. Generate only required embeddings
6. Apply changes to the in-memory FAISS index
7. Save the next FAISS generation
8. Commit chunk metadata and generation in SQLite
9. Make the new generation available to readers

If the process fails before Step 8, SQLite still references the previous generation. The unfinished snapshot can be discarded.

If the SQLite transaction succeeds, the new generation becomes authoritative. Readers must load the snapshot identified by SQLite rather than automatically selecting the newest file.

The earlier VectorStore constructor must therefore be adapted to load a specified generation. Its legacy vectors.faiss file should not be used as the authoritative index after this change.

Prevent Concurrent Writers

Generation checks detect conflicting commits, but they do not replace writer coordination. Two processes could otherwise attempt to create the same snapshot filename.

For this tutorial, we will maintain a single dedicated indexing worker and serialize indexing operations. This constraint must apply across all application instances, not merely within one Celery worker process.

A production deployment with multiple writers would require a distributed lock, database-backed work coordination, or a dedicated indexing service.

With versioned snapshots and transactional metadata updates, our incremental indexing for RAG architecture has a clear consistency model: SQLite identifies the committed FAISS generation, and every stored chunk references a stable vector ID within that generation.

Step 7 — Integrate Incremental Indexing with Celery

Our incremental indexing for RAG pipeline now has the components needed to detect document changes, compare chunks, and update FAISS selectively.

We will connect these components to Celery so that document processing runs asynchronously without blocking the FastAPI application.

Load the Committed FAISS Generation

Before integrating the pipeline, update VectorStore to load the generation recorded in SQLite instead of relying on the original vectors.faiss file.

Replace the index-loading logic in its constructor with:

def __init__(
    self,
    storage_dir="data/indexes",
    dimension=384,
    generation=0
):
    self.storage_dir = Path(storage_dir)
    self.storage_dir.mkdir(
        parents=True,
        exist_ok=True
    )

    self.dimension = dimension

    self.index_path = (
        self.storage_dir
        / f"vectors-{generation}.faiss"
    )

    if generation == 0:
        self.index = faiss.IndexIDMap2(
            faiss.IndexFlatIP(dimension)
        )
    else:
        self.index = faiss.read_index(
            str(self.index_path)
        )

        if (
            self.index.d != dimension
            or not isinstance(
                self.index,
                faiss.IndexIDMap2
            )
        ):
            raise ValueError(
                "Incompatible FAISS index"
            )

Keep the add_vectors(), remove_vectors(), search(), and save_generation() methods from the previous steps.

Also add a method to DocumentRegistry:

def get_generation(self) -> int:
    with self.connect() as connection:
        row = connection.execute(
            "SELECT generation FROM index_state WHERE id = 1"
        ).fetchone()

    return row[0]

Create the Incremental Indexer

Create ingestion/incremental_indexer.py. This module will coordinate the entire indexing operation.

from ingestion.hashing import (
    calculate_file_hash,
    calculate_chunk_hash
)
from ingestion.extractor import extract_text
from ingestion.cleaner import clean_text
from ingestion.chunker import create_document_chunks
from ingestion.chunk_diff import compare_chunks
from ingestion.embeddings import generate_embeddings
from ingestion.registry import DocumentRegistry
from ingestion.vector_store import VectorStore


def index_document(document_id: str, file_path: str):
    registry = DocumentRegistry()

    file_hash = calculate_file_hash(file_path)

    state = registry.detect_change(
        document_id,
        file_hash
    )

    if state == "UNCHANGED":
        return {
            "document_id": document_id,
            "status": "UNCHANGED",
            "embeddings_generated": 0
        }

    text = clean_text(
        extract_text(file_path)
    )

    if not text:
        raise ValueError("Document contains no text")

    chunks = create_document_chunks(
        document_id=document_id,
        text=text
    )

    old_records = registry.get_chunks(document_id)

    diff = compare_chunks(
        old_records,
        chunks
    )

    generation = registry.get_generation()

    store = VectorStore(
        generation=generation
    )

    new_ids = registry.allocate_vector_ids(
        len(diff.added)
    )

    if diff.added:
        embeddings = generate_embeddings(

[chunk.text for chunk in diff.added]

) else: embeddings = None removed_count = store.remove_vectors( diff.removed_ids ) if removed_count != len(diff.removed_ids): raise RuntimeError( “FAISS removal count mismatch” ) if diff.added: store.add_vectors( embeddings, new_ids ) chunk_records = [] for vector_id, chunk in diff.unchanged: chunk_records.append({ “vector_id”: vector_id, “chunk_hash”: calculate_chunk_hash( chunk.text ), “position”: chunk.position, “text”: chunk.text }) for vector_id, chunk in zip( new_ids, diff.added ): chunk_records.append({ “vector_id”: vector_id, “chunk_hash”: calculate_chunk_hash( chunk.text ), “position”: chunk.position, “text”: chunk.text }) new_generation = generation + 1 store.save_generation(new_generation) registry.commit_document_update( document_id=document_id, file_hash=file_hash, chunk_records=chunk_records, expected_generation=generation, new_generation=new_generation ) return { “document_id”: document_id, “status”: “INDEXED”, “chunks_reused”: len(diff.unchanged), “embeddings_generated”: len(diff.added), “vectors_removed”: removed_count, “generation”: new_generation }

The function skips unchanged documents and generates embeddings only for new or modified chunks. Existing vector IDs are preserved when their content remains unchanged.

This implementation assumes exclusive access to the indexing pipeline. It also assumes that the embedding model and preprocessing configuration have not changed since the previous generation.

Connect the Pipeline to Celery

In tasks/document_tasks.py, replace the previous document-processing function with our incremental indexer:

from tasks.celery_app import celery_app
from ingestion.incremental_indexer import index_document


@celery_app.task(
    name="rag.process_document",
    bind=True
)
def process_document_task(
    self,
    document_id: str,
    file_path: str
):
    return index_document(
        document_id=document_id,
        file_path=file_path
    )

The task name remains unchanged, so our existing FastAPI endpoint can continue submitting jobs through Celery.

However, document updates must reuse the original document_id. The upload endpoint from our previous tutorial generated a new UUID for every upload, so it must be extended to accept updates to an existing document.

Run a Dedicated Indexing Worker

Start the worker with a dedicated queue:

celery -A tasks.celery_app:celery_app worker \
  --queues=document_ingestion \
  --concurrency=1 \
  --loglevel=INFO

Run only one consumer of this queue across the deployment. Setting --concurrency=1 on multiple workers would not prevent concurrent index modifications.

Celery may redeliver tasks after failures. Repeated processing of an already committed document is safe when its hash and configuration match, because the indexer returns UNCHANGED.

For production use, retries must also account for interrupted snapshot creation and partially completed updates. Orphaned snapshots should be cleaned up only after confirming that SQLite does not reference them.

Keep Retrieval Workers Synchronized

An important detail in incremental indexing for RAG is that updating the index on disk does not automatically refresh FAISS instances already loaded by API workers.

Retrieval workers should check the committed generation in SQLite and reload the corresponding FAISS snapshot when it changes. A request should use one consistent snapshot and its matching metadata throughout retrieval.

With Celery integration complete, our RAG application can process document updates asynchronously while preserving embeddings for unchanged content.

Test Incremental Indexing

Now that our pipeline supports asynchronous document updates, we need to verify that incremental indexing for RAG actually reduces embedding generation without introducing inconsistencies between FAISS and SQLite.

We will test four scenarios: initial indexing, repeated ingestion, partial document updates, and chunk removal.

Test 1 — Index a New Document

Create a test document at data/uploads/test.txt containing several paragraphs about RAG architecture, embeddings, and vector search.

Run the indexer directly before testing the Celery integration:

from ingestion.incremental_indexer import index_document

result = index_document(
    document_id="test-001",
    file_path="data/uploads/test.txt"
)

print(result)

The first execution should return INDEXED. Because the document is new, every generated chunk requires an embedding.

Test 2 — Repeat Without Changes

Run the same operation again:

result = index_document(
    document_id="test-001",
    file_path="data/uploads/test.txt"
)

assert result["status"] == "UNCHANGED"
assert result["embeddings_generated"] == 0

This confirms that the document hash prevents unnecessary extraction, chunking, and embedding generation.

Test 3 — Modify One Paragraph

Edit one paragraph in the document and run the indexer again. The document hash should change, triggering chunk comparison.

Some chunks should retain their existing vector IDs, while the modified content receives new embeddings.

The exact number of changed chunks depends on the chunking algorithm. An inserted paragraph may affect neighboring chunk boundaries, so changing one paragraph does not necessarily mean that exactly one embedding will be regenerated.

Test 4 — Remove Document Content

Delete one or more paragraphs and repeat the indexing operation. The pipeline should remove obsolete vector IDs while preserving vectors for unchanged chunks.

To verify the final state, compare the IDs stored in SQLite with those present in the committed FAISS snapshot.

import faiss

from ingestion.registry import DocumentRegistry
from ingestion.vector_store import VectorStore


registry = DocumentRegistry()

generation = registry.get_generation()

store = VectorStore(
    generation=generation
)

with registry.connect() as connection:
    rows = connection.execute(
        "SELECT vector_id FROM chunks"
    ).fetchall()

registry_ids = {
    int(row[0]) for row in rows
}

faiss_ids = {
    int(value)
    for value in faiss.vector_to_array(
        store.index.id_map
    )
}

assert registry_ids == faiss_ids
assert store.index.ntotal == len(registry_ids)

print("FAISS and SQLite are synchronized")

This test checks the entire indexed collection, not just the updated document. Every FAISS vector must have a matching SQLite record, and every chunk record must reference an existing vector.

Test Interrupted Updates

Finally, test recovery by deliberately interrupting processing after a new FAISS snapshot is saved but before SQLite commits the new generation.

After restarting the application, SQLite should still reference the previous committed generation. The incomplete update must not become visible to retrieval requests.

This test also reveals an important limitation of our current implementation: an uncommitted snapshot can occupy the next generation filename and prevent a subsequent retry from saving that generation. The recovery procedure must identify and safely remove or replace orphaned snapshots before retrying.

A complete production test suite should also verify concurrent task submission, embedding model compatibility, duplicate chunk occurrences, and retrieval behavior after generation changes.

These tests establish whether incremental indexing for RAG preserves existing embeddings, updates changed content, and maintains consistent vector identifiers across document versions.

Production Considerations

Our incremental indexing for RAG implementation reduces unnecessary embedding generation and supports selective FAISS updates. However, deploying it in production requires additional safeguards for data consistency, concurrency, recovery, and operational monitoring.

Recover from Interrupted Updates

Versioned FAISS snapshots allow the application to retain the previously committed index when an update fails. However, an interrupted operation may leave an orphaned snapshot that SQLite does not reference.

Before retrying an update, the indexing service should inspect the committed generation, identify unreferenced snapshots, and safely clean them up while preserving any versions still used by active readers.

For stronger durability, snapshot files and their containing directories should be synchronized before committing the new generation in SQLite.

Prevent Concurrent Index Modifications

Our implementation assumes a single indexing writer. Running multiple Celery workers against the same local FAISS files can cause conflicting updates, even when each worker has a concurrency setting of one.

Production deployments should use a dedicated indexing service or enforce an exclusive lock covering the complete update operation, from reading the current generation through committing the next one.

Track Embedding and Chunking Versions

Matching chunk hashes does not guarantee that existing embeddings remain compatible with the current pipeline.

The registry should also record the embedding model, normalization settings, and preprocessing configuration. Changing these settings may require regenerating affected embeddings or rebuilding the index.

Synchronize Retrieval Workers

API workers must not continue searching an outdated FAISS snapshot indefinitely after a new generation is committed.

A practical approach is to check the active generation periodically and reload the corresponding snapshot when it changes. Each retrieval request should use a consistent index generation and matching metadata.

Monitor Indexing Performance

Useful production metrics include document processing time, the number of reused chunks, generated embeddings, removed vectors, failed updates, and index snapshot size.

These measurements help determine whether incremental indexing for RAG provides meaningful performance improvements as the document collection grows.

Finally, remember that IndexFlatIP performs exact search and keeps its vectors in memory. Incremental updates reduce indexing work, but they do not eliminate the memory and search costs associated with a growing vector collection.

Where to Go Next

With incremental indexing for RAG, our application can update a document collection without regenerating embeddings for every existing chunk. This builds on several important components of a production-oriented RAG architecture.

If you are building the system from scratch, start with How to Build a RAG System in Python. This tutorial introduces document retrieval, embeddings, vector search, and answer generation.

Next, explore How to Build a Document Ingestion Pipeline for RAG in Python. It explains how to extract document text, clean it, generate chunks and embeddings, and store them in FAISS.

To process document updates asynchronously, read How to Add Background Tasks to a RAG API with Celery and Redis. It covers background job execution, task queues, and job status tracking with FastAPI.

Together, these components provide a foundation for RAG applications that can ingest new documents, process updates in the background, and keep their searchable knowledge bases current.

As the collection grows, the next engineering challenge is to measure indexing efficiency, manage concurrent updates, and maintain reliable retrieval during index generation changes.

Frequently Asked Questions

What is incremental indexing in RAG?

Incremental indexing for RAG is a technique that updates only the changed portions of a document collection instead of reprocessing the entire knowledge base. It identifies new, modified, and deleted chunks while preserving embeddings for unchanged content.

Does incremental indexing improve RAG retrieval speed?

Not directly. Incremental indexing primarily reduces document processing time and embedding generation costs. Retrieval performance depends on the vector index type, collection size, search configuration, and available hardware.

Can FAISS update individual vectors?

FAISS supports adding and removing vectors when an appropriate index configuration is used. For example, IndexIDMap2 with IndexFlatIP supports explicit vector IDs and selective removal.

Replacing an existing vector can be implemented by removing its old ID and inserting a new embedding. These operations must be coordinated with the metadata store.

How does incremental indexing detect document changes?

Our implementation uses SHA-256 hashes at two levels. Document hashes identify unchanged source files, while chunk hashes identify text fragments whose existing embeddings can be reused.

Changes to the embedding model or preprocessing configuration must be tracked separately because identical text does not guarantee compatible embeddings.

What happens when a document is deleted?

The indexing pipeline must remove the document’s vectors from FAISS and delete its associated metadata records. Deletion should be performed through a coordinated update so that the committed index and registry remain consistent.

Document removal can be triggered explicitly or detected by comparing the registry with a complete, authoritative inventory of source documents.

Is incremental indexing suitable for production RAG systems?

Yes. Incremental indexing for RAG is particularly useful when documents change frequently or embedding generation is expensive.

However, a production implementation also needs reliable recovery procedures, exclusive writer coordination, embedding version tracking, consistent retrieval snapshots, and monitoring. The example developed in this tutorial provides the architectural foundation for these capabilities.

Conclusion

In this tutorial, we explored how to implement incremental indexing for RAG in Python using FAISS, Sentence Transformers, SQLite, and Celery.

We introduced document and chunk hashing, implemented change detection, and used explicit FAISS vector IDs to preserve unchanged embeddings while adding new vectors and removing outdated ones.

We also designed a versioned index persistence strategy to coordinate FAISS snapshots with SQLite metadata and integrated document updates into an asynchronous Celery processing pipeline.

The main benefit of incremental indexing for RAG is efficiency: as a knowledge base grows, the application can process document changes without regenerating embeddings for the entire collection.

Before production deployment, the implementation should be extended with complete recovery handling, concurrency protection, and embedding configuration versioning.

With these improvements, incremental indexing becomes an important building block for scalable RAG applications that rely on continuously updated documents.

Scroll to Top