How to Add Background Tasks to a RAG API with Celery and Redis

How to Add Background Tasks to a RAG API with Celery and Redis

Table of Contents

What You Will Build

In this tutorial, we will learn how to add background tasks in a RAG API using FastAPI, Celery, and Redis. We will extend the production-oriented RAG assistant developed in our previous tutorials by introducing a distributed task queue for operations that do not need to finish during an HTTP request.

Our existing application already supports document retrieval, reranking, LLM-generated answers, authentication, caching, monitoring, and horizontal scaling. However, some RAG operations can take significantly longer than a typical API request.

For example, uploading a large collection of documents may require text extraction, chunking, embedding generation, and vector index updates. Executing these operations directly inside an HTTP request can increase response times and consume resources needed for interactive questions.

From Synchronous Processing to Background Jobs

A traditional synchronous API waits until the requested operation finishes:

Client
   │
   ▼
FastAPI
   │
   ▼
Process Documents
   │
   ▼
Generate Embeddings
   │
   ▼
Update Vector Index
   │
   ▼
Return Response

For large document collections, this process may take minutes. The client must wait, and the application must manage a long-running request.

With background processing, FastAPI accepts the job and immediately returns a task identifier:

Client
   │
   ▼
FastAPI
   │
   ▼
Redis Queue
   │
   ▼
Celery Worker
   │
   ├── Extract Text
   ├── Create Chunks
   ├── Generate Embeddings
   └── Update Vector Index
   │
   ▼
Store Task Result

The client can check the task status through a separate API endpoint while the worker processes the documents independently.

Technology Stack

We will use three main components:

  • FastAPI — accepts requests, submits background jobs, and provides endpoints for checking task status.
  • Celery — executes background jobs using independent worker processes.
  • Redis — acts as the Celery message broker and stores task results.

Celery workers operate separately from FastAPI. This allows us to increase background processing capacity without increasing the number of HTTP API replicas.

What the Completed System Will Support

By the end of this tutorial, our application will support:

POST /jobs/documents
    │
    └── Submit document processing job
        Return task_id

GET /jobs/{task_id}
    │
    └── Check task status
        PENDING / STARTED / SUCCESS / FAILURE

Celery Workers
    │
    ├── Process documents
    ├── Generate embeddings
    ├── Update vector resources
    └── Store task results

We will also implement error handling, controlled retries, and a Docker Compose configuration that runs FastAPI, Redis, and Celery workers as separate services.

When Should You Use Background Tasks?

Not every RAG operation belongs in a queue. Interactive question answering should generally remain synchronous when users expect an immediate response.

Background processing is more appropriate for document ingestion, embedding generation, index rebuilding, batch evaluation, and other operations that can run independently of the original HTTP request.

This distinction is important when designing background tasks in a RAG API. Our goal is not to make every request asynchronous, but to separate interactive workloads from long-running operations and allow each to scale independently.

Next, we will review the existing project and identify the components required before introducing Celery.

Prerequisites

Before implementing background tasks in a RAG API, we need an existing FastAPI application with a working RAG pipeline. We will continue extending the same project used in our previous tutorials rather than building a separate application.

Our starting point is a Dockerized RAG API that supports document retrieval, LLM-generated answers, Redis caching, authentication, and monitoring. Celery will introduce a separate execution layer for long-running operations without changing how interactive questions are processed.

Existing RAG API

The application already exposes an endpoint for answering questions:

@app.post(
    "/ask",
    dependencies=[
        Depends(verify_api_key)
    ]
)
@limiter.limit(RAG_RATE_LIMIT)
async def ask(
    request: Request,
    body: AskRequest
):
    answer = ask_rag(
        body.question,
        request_id=request.state.request_id
    )

    return {
        "answer": answer
    }

This endpoint will remain available for interactive question answering. We will add separate endpoints for submitting and monitoring background jobs.

Required Python Packages

The existing project already uses FastAPI and Redis. We will add Celery and ensure that the Redis Python client is installed:

pip install "celery[redis]"

The Redis extra installs the dependencies Celery needs to communicate with Redis as a message broker and result backend.

Add the dependency to requirements.txt so Docker can install it when rebuilding the application image.

Existing Docker Environment

Our previous scaling tutorial introduced Docker Compose with FastAPI, Redis, and Nginx. For the background processing system, we will reuse the existing Redis service and add a dedicated Celery worker service.

Docker Compose
      │
      ├── Nginx
      │
      ├── FastAPI
      │
      ├── Redis
      │
      └── Celery Worker (new)

Redis will now serve two different purposes:

  • RAG caching — storing previously generated answers and other cached data.
  • Celery infrastructure — transporting queued tasks and storing their execution results.

We will use separate Redis databases for these responsibilities:

redis://redis:6379/0  → RAG Cache
redis://redis:6379/1  → Celery Broker
redis://redis:6379/2  → Celery Results

This provides logical separation between the different types of data. It does not provide resource isolation: all three databases still share the same Redis server, memory, and processing capacity.

Prepare the Project Structure

We will organize the new components as follows:

rag-project/
│
├── main.py
├── rag_pipeline.py
│
├── tasks/
│   ├── __init__.py
│   ├── celery_app.py
│   └── document_tasks.py
│
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
└── nginx.conf

The celery_app.py module will configure Celery, while document_tasks.py will contain background document-processing tasks.

The existing RAG pipeline will remain independent of the HTTP endpoints and Celery configuration. This separation allows both FastAPI and Celery workers to reuse application logic without duplicating it.

Document Processing Requirements

Our first background job will represent a document ingestion workflow:

Document
   │
   ▼
Text Extraction
   │
   ▼
Chunking
   │
   ▼
Embedding Generation
   │
   ▼
Vector Index Update

We will assume that the project already has reusable functions for document processing and embedding generation from the earlier RAG tutorials.

One important architectural requirement is that document files and vector resources must be accessible to the worker. A Celery worker runs in a separate process and potentially in a separate container, so it cannot rely on files stored only inside a FastAPI container.

For the initial implementation, we can use a shared Docker volume. Larger deployments may use object storage and an external vector database.

With these prerequisites in place, we can introduce background tasks in a RAG API without disrupting the existing question-answering workflow. Next, we will examine how Celery queues, workers, and task results interact.

Step 1 — Understand Background Tasks and Job Queues

Before writing Celery code, we need to understand how a task queue changes request processing. When implementing background tasks in a RAG API, the main objective is to separate the submission of a job from its execution.

FastAPI accepts the request, Celery manages task execution, and Redis transfers messages between the application and workers.

How a Task Queue Works

Consider a user who uploads a large PDF containing hundreds of pages. Extracting text, generating embeddings, and updating a vector index may take several minutes.

Instead of waiting for the entire operation, the client submits a job:

Client
   │
   │ POST /jobs/documents
   ▼
FastAPI
   │
   │ Publish Task
   ▼
Redis Broker
   │
   │ Deliver Task
   ▼
Celery Worker
   │
   │ Process Document
   ▼
Redis Result Backend

FastAPI returns a task identifier immediately after the broker accepts the task:

{
    "task_id": "a84d9e2f-...",
    "status": "PENDING"
}

The client can later use this identifier to retrieve the task status and result.

Understand the Three Main Components

FastAPI acts as the task producer. It validates incoming requests and publishes task messages. It does not execute the document-processing operation itself.

Redis acts as the message broker. It stores messages until Celery workers receive them. The broker is responsible for delivering tasks, not for executing Python functions.

Celery workers act as task consumers. They receive messages, execute the corresponding Python functions, and report execution results.

We will also configure Redis as the result backend. Although the broker and result backend use the same Redis server in our example, they perform different functions.

Understand Task States

Celery provides several task states:

StateMeaning
PENDINGThe result backend has no recorded state for this task.
STARTEDA worker has started executing the task.
RETRYThe task is waiting for another execution attempt.
SUCCESSThe task completed successfully.
FAILUREThe task failed.

The STARTED state requires task-start tracking to be enabled. We will configure this explicitly when creating the Celery application.

Also, PENDING does not necessarily mean that a task is waiting in the queue. It can indicate an unknown task identifier or a result that has expired. A production API should distinguish these cases using its own job records when necessary.

Background Tasks Are Not the Same as Async Functions

FastAPI supports asynchronous endpoints using async def, but asynchronous programming does not automatically move CPU-intensive work into a separate process.

For example, generating embeddings with a local model may consume substantial CPU or GPU resources. Running this operation directly inside an asynchronous endpoint can still affect API performance.

Celery provides a different execution model:

FastAPI Process
      │
      │ Submit Job
      ▼
Redis Queue
      │
      ▼
Separate Worker Process
      │
      ▼
CPU / GPU Intensive Work

The worker can run in another container or even on another machine, allowing us to allocate resources independently.

Understand Task Delivery and Retries

A task queue does not guarantee that every operation executes exactly once. Depending on acknowledgement settings, failures, and worker interruptions, a task may be executed again.

This matters for document ingestion. Repeating the same task should not create duplicate document records or insert identical chunks into the vector index multiple times.

We should therefore design document-processing tasks to be idempotent: repeating a task with the same document identifier should produce the intended final state without unwanted duplication.

Retries also need to distinguish temporary errors, such as an unavailable external embedding service, from permanent errors, such as an unsupported or corrupted document.

When Is a Task Queue Worth Using?

A queue introduces additional infrastructure and operational complexity. It is most useful when jobs are long-running, resource-intensive, or need independent retry and scaling policies.

For our project, document ingestion is a suitable first use case. Interactive question answering will continue using the existing /ask endpoint.

This separation is the central architectural principle behind background tasks in a RAG API: HTTP requests remain responsive while independent workers handle expensive operations.

Next, we will configure Celery with Redis and create the application object shared by FastAPI and the worker processes.

Step 2 — Install and Configure Celery with Redis

Now that we understand how task queues work, we can configure Celery for our existing FastAPI application. This is the foundation for implementing background tasks in a RAG API without executing long-running operations inside HTTP requests.

We will use Redis as both the message broker and the result backend, with separate logical databases for each responsibility.

Install Celery

Install Celery with Redis support:

pip install "celery[redis]"

Add the dependency to requirements.txt. For reproducible deployments, use a tested, pinned version rather than leaving the dependency unbounded.

Configure the Celery Application

Create tasks/celery_app.py:

import os

from celery import Celery


REDIS_BROKER_URL = os.getenv(
    "CELERY_BROKER_URL",
    "redis://localhost:6379/1"
)

REDIS_RESULT_URL = os.getenv(
    "CELERY_RESULT_BACKEND",
    "redis://localhost:6379/2"
)


celery_app = Celery(
    "rag_tasks",
    broker=REDIS_BROKER_URL,
    backend=REDIS_RESULT_URL,
    include=[
        "tasks.document_tasks"
    ]
)


celery_app.conf.update(
    task_track_started=True,
    task_serializer="json",
    result_serializer="json",
    accept_content=["json"],
    result_expires=3600,
    task_time_limit=1800,
    task_soft_time_limit=1500,
    worker_prefetch_multiplier=1
)

Let’s examine the important settings.

For a detailed explanation of task configuration, message brokers, result backends, and worker settings, see the official Celery configuration documentation.

Broker and Result Backend

The broker transports task messages from FastAPI to Celery workers:

FastAPI
   │
   ▼
Redis DB 1
   │
   ▼
Celery Worker

The result backend stores task states and results:

Celery Worker
   │
   ▼
Redis DB 2
   │
   ▼
FastAPI Status Endpoint

The two Redis databases provide logical separation, although they still share the same Redis server resources.

Task Configuration

The task_track_started option allows Celery to report when a worker begins executing a task.

JSON serialization ensures that task messages and results use a predictable data format. It also avoids relying on Python pickle serialization for task communication.

The result_expires setting removes stored task results after one hour. This helps prevent old results from accumulating indefinitely.

The time limits protect workers from tasks that run excessively long. In this example, the soft limit is 1,500 seconds and the hard limit is 1,800 seconds. These values should be adjusted according to the expected document-processing workload.

Finally, worker_prefetch_multiplier=1 reduces how many tasks each worker process reserves in advance. This is useful when document-processing jobs have significantly different execution times.

Configure Environment Variables

Inside Docker Compose, Redis is accessible through its service name. We will provide the following variables to both FastAPI and Celery:

environment:
  CELERY_BROKER_URL: redis://redis:6379/1
  CELERY_RESULT_BACKEND: redis://redis:6379/2

The existing RAG cache continues using:

REDIS_URL: redis://redis:6379/0

For local development outside Docker, the default Celery configuration uses localhost. When running inside containers, the environment variables override those defaults.

Verify the Celery Configuration

With Redis running locally, start a Celery worker from the project root:

celery -A tasks.celery_app:celery_app worker \
  --loglevel=INFO

Celery will load the application configuration, connect to Redis, and attempt to register the tasks listed in include.

At this stage, tasks.document_tasks has not been implemented yet. Create an empty document_tasks.py file before starting the worker, or wait until the next step to run this command.

The worker should report that it is connected to Redis and ready to receive tasks. On Windows development machines, running Celery workers inside Linux containers or WSL is preferable because Celery’s standard prefork execution model is designed for Unix-like environments.

Keep Task Configuration Separate from RAG Logic

The Celery configuration should not contain document-processing or retrieval logic. Its responsibility is to define how tasks are registered, transported, executed, and tracked.

This separation makes background tasks in a RAG API easier to maintain. FastAPI can submit jobs, Celery workers can execute them, and both components can reuse the same document-processing functions.

Next, we will implement our first Celery task for document ingestion, including text processing, embedding generation, and vector index updates.

Step 3 — Create Background RAG Tasks

Now we can implement our first Celery task. We will use document ingestion as a practical example because extracting text, generating embeddings, and updating vector indexes can take considerable time.

When implementing background tasks in a RAG API, it is important to keep the processing logic independent of Celery. The task should coordinate existing functions rather than contain the entire document-processing pipeline.

Separate Document Processing from Task Execution

Our RAG project already contains components for chunking, embedding generation, and vector search. We will reuse those components through a dedicated document-processing function.

Create document_processor.py:

from pathlib import Path

from rag_pipeline import (
    extract_text,
    chunk_text,
    generate_embeddings,
    update_vector_index
)


def process_document(
    document_id: str,
    file_path: str
):
    path = Path(file_path)

    if not path.is_file():
        raise FileNotFoundError(
            f"Document not found: {file_path}"
        )

    text = extract_text(path)

    chunks = chunk_text(text)

    if not chunks:
        raise ValueError(
            "Document contains no usable text"
        )

    embeddings = generate_embeddings(chunks)

    update_vector_index(
        document_id=document_id,
        chunks=chunks,
        embeddings=embeddings
    )

    return {
        "document_id": document_id,
        "chunks_processed": len(chunks)
    }

The imported functions represent the document-processing interfaces from our existing RAG project. Their exact signatures may differ depending on the implementation developed in earlier tutorials.

The important point is that process_document() does not depend on FastAPI or Celery. It can be executed directly during testing or called from a background worker.

Create the Celery Task

Open tasks/document_tasks.py:

from celery.exceptions import (
    SoftTimeLimitExceeded
)

from tasks.celery_app import celery_app
from document_processor import process_document


@celery_app.task(
    bind=True,
    name="rag.process_document",
    max_retries=3
)
def process_document_task(
    self,
    document_id: str,
    file_path: str
):
    try:
        result = process_document(
            document_id=document_id,
            file_path=file_path
        )

        return result

    except SoftTimeLimitExceeded:
        raise

    except (FileNotFoundError, ValueError):
        raise

    except Exception as exc:
        raise self.retry(
            exc=exc,
            countdown=30
        )

The @celery_app.task decorator registers the function as a Celery task.

The bind=True option gives the function access to the task instance through self. This allows us to call self.retry() when an operation fails.

We also assign an explicit task name, rag.process_document, so the producer and worker can refer to the same registered task.

Understand Retry Behavior

The task distinguishes between errors that should not normally be retried and errors that might be temporary.

Document Processing
        │
        ├── Success
        │      └── Return Result
        │
        ├── Invalid Document
        │      └── FAILURE
        │
        └── Unexpected Error
               │
               ▼
             RETRY
               │
               ▼
         Execute Again

For example, a missing document or a file containing no usable text will fail immediately.

Other exceptions trigger a retry after 30 seconds, up to three retries.

This broad retry policy is suitable for demonstrating Celery’s behavior, but production applications should classify exceptions more precisely. A programming error or invalid model configuration should not be retried repeatedly.

Pass References Instead of Large Documents

Notice that our task accepts only two arguments:

document_id: str
file_path: str

We do not send the document contents, embeddings, or vector arrays through Redis.

Task messages should remain small. Large payloads increase broker memory usage, network traffic, and serialization overhead.

Instead, FastAPI will store the uploaded document in a location accessible to the worker and pass a reference to that file.

FastAPI
   │
   ├── Save Document
   │       │
   │       ▼
   │   Shared Storage
   │
   └── Submit Task
           │
           ▼
       Redis Broker
           │
           ▼
       Celery Worker
           │
           ▼
       Read Document
           │
           ▼
       Process Document

In our Docker Compose implementation, we will use a shared volume. In a distributed production environment, object storage is often a more appropriate choice.

Prevent Duplicate Index Updates

Celery tasks can be retried or executed again after certain failures. This creates an important requirement for document ingestion: processing the same document twice should not produce duplicate vector records.

A robust indexing operation should use a stable document identifier and replace or upsert the corresponding records.

Document ID: doc_123
        │
        ▼
Generate Chunks
        │
        ▼
Replace Existing Records
        │
        ▼
Store New Index Version

There is another important detail. Our earlier RAG tutorials used FAISS as a local vector index. If a Celery worker updates its own in-memory FAISS index, the FastAPI replicas will not automatically see those changes.

For this tutorial, update_vector_index() must therefore publish updates through a storage mechanism shared with the API. One approach is to build a new immutable index version and have API replicas reload it safely. Another is to use a dedicated vector database that supports concurrent updates.

Simply writing to an index inside the worker process is not sufficient.

Keep the Task Result Small

After successful execution, Celery stores a compact result:

{
    "document_id": "doc_123",
    "chunks_processed": 148
}

The result confirms that processing completed. It does not contain the extracted document text or generated embeddings.

This keeps Redis result storage manageable and allows the client to retrieve the processing outcome without transferring unnecessary data.

We now have the core processing component needed for background tasks in a RAG API. The document-processing function performs the actual work, while Celery manages execution, task states, and retries.

Next, we will connect this task to FastAPI by creating an endpoint that accepts a document, saves it to shared storage, submits the Celery job, and immediately returns a task identifier.

Step 4 — Submit Background Tasks Through FastAPI

Now that our Celery task is ready, we need an API endpoint that accepts documents and submits processing jobs. This is where background tasks in a RAG API become accessible to external clients.

The endpoint will save an uploaded document to shared storage, create a unique document identifier, submit the Celery task, and return a response without waiting for document processing to finish.

Create the Document Upload Endpoint

FastAPI supports file uploads through UploadFile. Install the required dependency:

pip install python-multipart

Add python-multipart to requirements.txt.

Create document_api.py:

import os
import uuid
import shutil

from pathlib import Path

from fastapi import (
    APIRouter,
    UploadFile,
    File,
    HTTPException,
    Depends,
    status
)

from tasks.document_tasks import (
    process_document_task
)

from auth import verify_api_key


router = APIRouter(
    prefix="/jobs",
    tags=["Background Jobs"]
)

UPLOAD_DIR = Path(
    os.getenv(
        "UPLOAD_DIR",
        "./data/uploads"
    )
)

UPLOAD_DIR.mkdir(
    parents=True,
    exist_ok=True
)


@router.post(
    "/documents",
    status_code=status.HTTP_202_ACCEPTED,
    dependencies=[
        Depends(verify_api_key)
    ]
)
def submit_document(
    file: UploadFile = File(...)
):
    document_id = str(uuid.uuid4())

    suffix = Path(
        file.filename or ""
    ).suffix.lower()

    if suffix not in {".pdf", ".txt"}:
        raise HTTPException(
            status_code=400,
            detail="Unsupported file type"
        )

    file_path = (
        UPLOAD_DIR /
        f"{document_id}{suffix}"
    )

    try:
        with file_path.open("wb") as output:
            shutil.copyfileobj(
                file.file,
                output
            )

        task = process_document_task.delay(
            document_id,
            str(file_path)
        )

    except Exception:
        file_path.unlink(
            missing_ok=True
        )

        raise HTTPException(
            status_code=503,
            detail="Unable to submit document job"
        )

    finally:
        file.file.close()

    return {
        "task_id": task.id,
        "document_id": document_id,
        "status": "PENDING"
    }

The authentication dependency follows the same approach used in our existing RAG API. Adjust the auth import to match the authentication module in your project.

The endpoint accepts PDF and TXT files, but the extension check is only a basic validation step. Production applications should also enforce upload size limits, inspect file contents, and apply appropriate security checks before processing untrusted documents.

Register the Router

Open main.py and register the new router:

from fastapi import FastAPI

from document_api import router as document_router


app = FastAPI()

app.include_router(
    document_router
)

In the existing project, keep the original FastAPI application and its middleware, authentication, monitoring, and RAG endpoints. Only add the router import and include_router() call.

Submit a Document

Start FastAPI and the Celery worker, then submit a document:

curl -X POST \
  http://localhost:8000/jobs/documents \
  -H "X-API-Key: your-api-key" \
  -F "file=@document.pdf"

FastAPI saves the uploaded file and publishes a message to Redis. The response looks like this:

{
    "task_id": "b6f9d8a2-...",
    "document_id": "a12c47e9-...",
    "status": "PENDING"
}

The HTTP status code is 202 Accepted. This means the API accepted the job for asynchronous processing, not that document ingestion has completed successfully.

Understand the Request Lifecycle

POST /jobs/documents
        │
        ▼
Authenticate Request
        │
        ▼
Validate File Extension
        │
        ▼
Generate Document ID
        │
        ▼
Save File
        │
        ▼
Submit Celery Task
        │
        ▼
Return HTTP 202
        │
        ▼
Client Receives Task ID

Meanwhile, a Celery worker receives the task and begins processing the document independently.

The client does not need to maintain an open HTTP connection while embeddings are generated or the vector index is updated.

Why Use a Unique Document Identifier?

Each uploaded document receives a UUID. This identifier separates document identity from the original filename and helps prevent filename collisions.

For example, two users can upload files named report.pdf without overwriting each other’s documents.

The document identifier also provides a stable reference for index updates, document replacement, and future deletion operations.

Keep Uploaded Files Accessible to Workers

The most important deployment requirement is shared storage.

FastAPI Container
       │
       ▼
Shared Upload Volume
       │
       ▼
Celery Container

If FastAPI saves the file only inside its own container filesystem, a separate Celery container will not be able to read it.

In Step 7, we will mount the same Docker volume into both services and configure a common UPLOAD_DIR.

For deployments across multiple machines, shared object storage is generally more suitable than relying on local filesystem paths.

Important Production Limitations

This implementation establishes the basic job-submission workflow. However, several additional safeguards are needed before exposing document ingestion to untrusted users.

Uploads should have strict size limits, supported file formats should be validated beyond their extensions, and files should be associated with the authenticated user. The API should also prevent clients from accessing jobs or documents belonging to other users.

Another consideration is the boundary between saving a file and publishing its task. These are separate operations. A production system may need persistent job records and a recovery mechanism to handle failures between storage and queue submission.

With this endpoint, we have connected FastAPI to Celery and established the submission mechanism for background tasks in a RAG API.

Next, we will create an endpoint that allows clients to check whether a job is pending, running, completed, or failed, and retrieve its result.

Step 5 — Track Task Status and Results

After submitting a document, the client receives a task identifier but does not know when processing will finish. To make background tasks in a RAG API practical, we need an endpoint that reports task status and returns the result after successful execution.

Celery provides the AsyncResult interface for retrieving task information from the configured result backend. We can expose this functionality through FastAPI.

Create a Task Status Endpoint

Open document_api.py and add the following imports:

from celery.result import AsyncResult

from tasks.celery_app import celery_app

Then add a new endpoint to the existing router:

@router.get(
    "/{task_id}",
    dependencies=[
        Depends(verify_api_key)
    ]
)
def get_task_status(task_id: str):

    task = AsyncResult(
        task_id,
        app=celery_app
    )

    response = {
        "task_id": task_id,
        "status": task.state
    }

    if task.successful():
        response["result"] = task.result

    elif task.failed():
        response["error"] = (
            "Document processing failed"
        )

    return response

The endpoint retrieves the current state of a task and returns its result only after successful completion.

For failed tasks, we return a generic error message instead of exposing internal exception details. Detailed error information should remain available through application logs and monitoring.

Check the Status of a Submitted Job

After submitting a document, the client can query its task identifier:

curl \
  http://localhost:8000/jobs/YOUR_TASK_ID \
  -H "X-API-Key: your-api-key"

While the task is waiting or running, the response may look like this:

{
    "task_id": "b6f9d8a2-...",
    "status": "STARTED"
}

After successful processing:

{
    "task_id": "b6f9d8a2-...",
    "status": "SUCCESS",
    "result": {
        "document_id": "a12c47e9-...",
        "chunks_processed": 148
    }
}

If the task fails:

{
    "task_id": "b6f9d8a2-...",
    "status": "FAILURE",
    "error": "Document processing failed"
}

Understand Task State Transitions

A successful task generally follows this sequence:

PENDING
   │
   ▼
STARTED
   │
   ▼
SUCCESS

A task that encounters a temporary error may enter the retry state:

PENDING
   │
   ▼
STARTED
   │
   ▼
RETRY
   │
   ▼
STARTED
   │
   ├── SUCCESS
   │
   └── FAILURE

These diagrams show typical transitions, not every possible execution path. For example, tasks can fail before a STARTED state is observed.

Poll for Task Completion

A client can periodically check the status endpoint until the task reaches a terminal state.

For example, a simple Python client can poll every two seconds:

import time
import requests


TASK_ID = "your-task-id"

URL = (
    f"http://localhost:8000/jobs/{TASK_ID}"
)

HEADERS = {
    "X-API-Key": "your-api-key"
}


while True:

    response = requests.get(
        URL,
        headers=HEADERS,
        timeout=10
    )

    response.raise_for_status()

    data = response.json()

    print(data["status"])

    if data["status"] in {
        "SUCCESS",
        "FAILURE"
    }:
        print(data)
        break

    time.sleep(2)

This approach is straightforward and works well for relatively small applications. For large numbers of simultaneous jobs, polling frequency should be controlled to avoid unnecessary API and Redis traffic.

Understand the Limitations of Celery Results

Celery’s result backend is useful for tracking execution, but it is not a complete job management database.

In our configuration, task results expire after one hour:

result_expires=3600

After expiration, the result backend may report PENDING for a task that previously completed.

An unknown task identifier can also produce the same state. Therefore, PENDING should not be interpreted as proof that a job exists and is waiting for execution.

For production systems, a separate database table can maintain persistent job records:

jobs
│
├── job_id
├── task_id
├── document_id
├── user_id
├── status
├── created_at
├── completed_at
└── error_message

This allows the API to distinguish unknown jobs from queued jobs, preserve execution history, and enforce access permissions.

Our current endpoint requires authentication, but it does not yet verify that the authenticated user owns the requested task. That authorization check must be added before exposing job results in a multi-user production environment.

Keep Task Results Separate from Document Data

The result backend should store compact execution metadata rather than large documents or embedding arrays.

Celery Result
     │
     ├── Task Status
     ├── Document ID
     └── Processing Statistics

Shared Storage
     │
     ├── Original Documents
     └── Processed Data

Vector Storage
     │
     └── Document Embeddings

This separation prevents Redis result storage from growing unnecessarily and keeps document data available independently of task result expiration.

We now have the basic lifecycle for background tasks in a RAG API: submit a job, receive its identifier, monitor execution, and retrieve the result.

Next, we will improve reliability by handling task failures, configuring retries, and preventing repeated execution from creating duplicate document records.

Step 6 — Handle Task Failures and Retries

Document processing involves several operations that can fail independently: reading files, generating embeddings, accessing external services, and updating vector indexes. Reliable background tasks in a RAG API must distinguish temporary failures from permanent errors and prevent retries from corrupting stored data.

Classify Errors Before Retrying

Not every exception should trigger another execution attempt. A temporary network failure may disappear after a few seconds, while a corrupted document will usually fail again regardless of how many times we retry.

ErrorRecommended Action
Temporary network failureRetry with backoff
External service unavailableRetry with backoff
Embedding API rate limitRetry after an appropriate delay
Unsupported document formatFail immediately
Corrupted documentFail immediately
Invalid application configurationFail and alert the operator

In Step 3, our initial implementation retried almost every unexpected exception. We can improve that behavior by explicitly identifying recoverable failures.

Define a Retryable Exception

Create a small exception class in document_processor.py:

class TemporaryProcessingError(Exception):
    """A recoverable document processing error."""
    pass

The document-processing layer should raise this exception when a failure is known to be temporary. For example, an embedding service adapter can translate a network timeout or a temporary service-unavailable response into TemporaryProcessingError.

Permanent validation errors should continue raising exceptions such as ValueError.

Configure Automatic Retries

Replace the task implementation in tasks/document_tasks.py:

from celery.exceptions import SoftTimeLimitExceeded

from tasks.celery_app import celery_app

from document_processor import (
    process_document,
    TemporaryProcessingError
)


@celery_app.task(
    bind=True,
    name="rag.process_document",
    autoretry_for=(TemporaryProcessingError,),
    retry_backoff=True,
    retry_backoff_max=300,
    retry_jitter=True,
    max_retries=3
)
def process_document_task(
    self,
    document_id: str,
    file_path: str
):
    try:
        return process_document(
            document_id=document_id,
            file_path=file_path
        )

    except SoftTimeLimitExceeded:
        raise

    except Exception:
        raise

The autoretry_for option tells Celery to retry only when TemporaryProcessingError is raised.

The retry configuration provides exponential backoff, limits the maximum delay to 300 seconds, and introduces random jitter to reduce the chance of multiple workers retrying simultaneously.

With max_retries=3, Celery allows up to three retries after the initial execution attempt.

The explicit try block is not necessary here because both exception handlers simply re-raise their exceptions. The task can therefore be simplified:

@celery_app.task(
    bind=True,
    name="rag.process_document",
    autoretry_for=(TemporaryProcessingError,),
    retry_backoff=True,
    retry_backoff_max=300,
    retry_jitter=True,
    max_retries=3
)
def process_document_task(
    self,
    document_id: str,
    file_path: str
):
    return process_document(
        document_id=document_id,
        file_path=file_path
    )

This is the version we will keep in our project.

Prevent Duplicate Document Processing

Retries introduce another important problem. A task may successfully complete some operations before an error occurs.

Extract Text         SUCCESS
      │
      ▼
Generate Embeddings  SUCCESS
      │
      ▼
Update Vector Index  PARTIAL FAILURE
      │
      ▼
Retry Entire Task

If the task simply appends vectors during every execution, repeated attempts may create duplicate records.

A safer approach is to use the stable document_id as the identity of the indexing operation.

document_id = doc_123
         │
         ▼
Build New Index Data
         │
         ▼
Validate New Data
         │
         ▼
Replace Previous Version
         │
         ▼
Publish Updated Index

For a local FAISS implementation, this may involve building a new index version and publishing it atomically rather than modifying a shared index file in place.

For a vector database, stable chunk identifiers and supported upsert operations can help prevent duplicate records.

Idempotency must be implemented in the storage layer. Celery retries alone cannot guarantee it.

Understand Worker Failures

A worker can terminate unexpectedly while processing a document. This is different from a Python exception handled by Celery.

By default, Celery normally acknowledges tasks before executing them. Therefore, a worker failure can result in an interrupted task that is not automatically executed again.

Celery supports late acknowledgements:

task_acks_late=True

With late acknowledgements, tasks are acknowledged after execution rather than before it. However, this does not guarantee redelivery after every type of worker failure. Celery has additional behavior for terminated worker processes, and broker visibility timeouts must also be considered.

For Redis, tasks that remain unacknowledged beyond the configured visibility timeout may be delivered again. If the timeout is shorter than the actual processing time, duplicate execution can occur.

We will keep the default acknowledgement behavior in this introductory implementation. Before enabling late acknowledgements in production, ensure that document ingestion is idempotent and test worker termination and task redelivery explicitly.

Record Failures Without Exposing Internal Details

Our status endpoint already returns a generic message for failed jobs:

{
    "task_id": "b6f9d8a2-...",
    "status": "FAILURE",
    "error": "Document processing failed"
}

The client does not need to receive internal stack traces, storage paths, or service credentials.

For debugging and monitoring, Celery workers should log the task identifier, document identifier, exception type, and execution duration.

In a production system, persistent job records can also store a sanitized error category and completion timestamp independently of Celery’s expiring result backend.

Set Realistic Execution Limits

Our Celery configuration currently defines:

task_soft_time_limit=1500
task_time_limit=1800

These limits prevent tasks from running indefinitely, but they should reflect the expected workload.

A document containing ten pages and a collection containing several thousand pages should not necessarily use the same execution policy.

For large ingestion workloads, splitting processing into smaller independent jobs can improve retry efficiency and reduce the amount of work lost when a task fails.

Reliability Requires More Than Retries

Retries are only one part of reliable background processing. A complete system also needs durable document storage, idempotent index updates, controlled task execution, monitoring, and recovery procedures.

These mechanisms allow background tasks in a RAG API to recover from temporary failures without creating inconsistent document records or repeatedly executing permanently invalid jobs.

Next, we will configure Docker Compose to run FastAPI, Redis, and Celery workers as separate services, with shared document storage and independent worker scaling.

Step 7 — Run Celery Workers with Docker Compose

We now have a FastAPI endpoint for submitting document-processing jobs, a Celery task for executing them, and Redis for message delivery and result storage. The next step is to run these components as separate containers.

Docker Compose makes it possible to deploy background tasks in a RAG API while keeping HTTP request handling and document processing independent. We can also increase the number of Celery workers without changing the number of FastAPI replicas.

Reuse the Existing Docker Image

Our FastAPI application and Celery workers use the same Python modules, dependencies, and document-processing functions. Therefore, we can build both services from the existing Dockerfile.

The important difference is the command executed when each container starts:

FastAPI Container
    └── uvicorn main:app

Celery Container
    └── celery worker

The API accepts requests and submits jobs. Celery workers consume jobs from Redis and execute them independently.

Update Docker Compose

Extend the existing docker-compose.yml:

services:

  nginx:
    image: nginx:alpine
    ports:
      - "8000:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - rag-api

  rag-api:
    build: .
    command: >
      uvicorn main:app
      --host 0.0.0.0
      --port 8000

    environment:
      REDIS_URL: redis://redis:6379/0
      CELERY_BROKER_URL: redis://redis:6379/1
      CELERY_RESULT_BACKEND: redis://redis:6379/2
      UPLOAD_DIR: /app/data/uploads

    volumes:
      - uploads:/app/data/uploads

    depends_on:
      - redis

  celery-worker:
    build: .
    command: >
      celery
      -A tasks.celery_app:celery_app
      worker
      --loglevel=INFO
      --concurrency=2

    environment:
      REDIS_URL: redis://redis:6379/0
      CELERY_BROKER_URL: redis://redis:6379/1
      CELERY_RESULT_BACKEND: redis://redis:6379/2
      UPLOAD_DIR: /app/data/uploads

    volumes:
      - uploads:/app/data/uploads

    depends_on:
      - redis

  redis:
    image: redis:7-alpine
    restart: unless-stopped

volumes:
  uploads:

This configuration adds the celery-worker service while preserving the existing Nginx, FastAPI, and Redis architecture.

Understand Shared Document Storage

Both FastAPI and Celery mount the same named Docker volume:

volumes:
  - uploads:/app/data/uploads

When FastAPI saves a document, the Celery worker can read it using the same filesystem path.

FastAPI
   │
   │ Save document.pdf
   ▼
Docker Volume: uploads
   │
   │ Read document.pdf
   ▼
Celery Worker
   │
   ▼
Process Document

This works for containers running on the same Docker host. A standard local Docker volume does not automatically provide shared storage across multiple physical machines.

For multi-host deployments, use an appropriate shared storage system or object storage service.

The same consideration applies to vector indexes. The uploads volume shares source documents, but it does not automatically synchronize FAISS indexes between workers and API replicas. Index publication must use the shared storage or vector service strategy discussed earlier.

Start the Complete System

Build and start the containers:

docker compose up -d --build

Check the running services:

docker compose ps

You should see the API, Celery worker, Redis, and Nginx containers.

To inspect Celery startup and task execution, run:

docker compose logs -f celery-worker

The worker should connect to Redis and register the rag.process_document task.

Scale Celery Workers Independently

One of the main advantages of Celery is that background processing capacity can grow independently from HTTP API capacity.

Start three Celery worker containers:

docker compose up -d \
  --scale celery-worker=3

The architecture becomes:

                  Nginx
                    │
                    ▼
                  FastAPI
                    │
                    ▼
                Redis Broker
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
       Worker 1  Worker 2  Worker 3
          │         │         │
          └─────────┼─────────┘
                    ▼
              Shared Storage

Each worker container runs with --concurrency=2, allowing up to two tasks to execute concurrently within that worker under Celery’s default prefork execution model.

With three containers, this configuration provides up to six concurrent task execution slots.

However, six concurrent tasks do not necessarily provide six times the processing throughput. Embedding models, CPU resources, GPU memory, and external API limits can constrain performance.

Choose Worker Concurrency Carefully

Document ingestion often includes CPU-intensive operations such as text processing and local embedding generation.

For CPU-bound workloads, increasing concurrency beyond available processing capacity can reduce efficiency.

For GPU-backed embedding models, multiple worker processes may load separate model copies into GPU memory. In this case, lower concurrency or a dedicated inference service may be more appropriate.

Start with a small worker count and measure task throughput, execution time, CPU utilization, and memory consumption before increasing capacity.

Keep API and Worker Scaling Separate

Our architecture now supports two independent scaling dimensions:

Interactive Traffic
        │
        ▼
   FastAPI Replicas
        │
        ▼
   Answer Questions


Document Ingestion
        │
        ▼
   Redis Task Queue
        │
        ▼
   Celery Workers
        │
        ▼
   Process Documents

If interactive traffic increases, we can add FastAPI replicas. If the document-processing queue grows, we can add Celery workers.

This separation is one of the primary reasons to introduce background tasks in a RAG API. Expensive ingestion operations no longer need to compete directly with HTTP request handling inside the same application processes.

Next, we will test the complete workflow: upload a document, verify that Celery receives the task, monitor its status, and confirm that the processed document becomes available for retrieval.

Test the Background Task System

Our FastAPI, Celery, and Redis services are now connected. Before moving to production, we need to verify that documents are accepted, queued, processed, and indexed correctly. Testing background tasks in a RAG API requires checking the entire workflow, not just confirming that the containers are running.

Verify the Running Services

Start the application:

docker compose up -d --build

Check the containers:

docker compose ps

Then inspect the Celery worker logs:

docker compose logs --tail=50 celery-worker

The worker should connect to Redis and register the rag.process_document task. If the task is missing, verify that tasks.document_tasks is included in the Celery configuration.

Submit a Test Document

Create a small text file named test-document.txt:

Retrieval-Augmented Generation combines
information retrieval with language models.

A RAG system retrieves relevant documents
before generating an answer.

Vector search uses embeddings to find
semantically similar content.

Submit the document through FastAPI:

curl -X POST \
  http://localhost:8000/jobs/documents \
  -H "X-API-Key: your-api-key" \
  -F "file=@test-document.txt"

The API should return HTTP 202 Accepted with a task identifier:

{
    "task_id": "8e42c1f7-...",
    "document_id": "d9a7b3e2-...",
    "status": "PENDING"
}

This confirms that FastAPI accepted the document and submitted the task. It does not yet confirm that document processing succeeded.

Monitor Task Execution

Open another terminal and follow the worker logs:

docker compose logs -f celery-worker

When the worker receives the job, Celery should log the task identifier and execution activity.

The processing workflow is:

FastAPI
   │
   ▼
Save Document
   │
   ▼
Redis Broker
   │
   ▼
Celery Worker
   │
   ├── Extract Text
   ├── Create Chunks
   ├── Generate Embeddings
   └── Update Vector Index
   │
   ▼
Redis Result Backend

Check the Task Result

Use the task identifier returned by the upload endpoint:

curl \
  http://localhost:8000/jobs/YOUR_TASK_ID \
  -H "X-API-Key: your-api-key"

While the task is executing, the response may contain:

{
    "task_id": "8e42c1f7-...",
    "status": "STARTED"
}

After successful processing:

{
    "task_id": "8e42c1f7-...",
    "status": "SUCCESS",
    "result": {
        "document_id": "d9a7b3e2-...",
        "chunks_processed": 3
    }
}

The number of chunks depends on the chunking configuration. The example above is illustrative; the actual result may contain a different value.

Verify That the Document Is Searchable

A successful Celery task does not necessarily mean that the document is available to the RAG API.

If the worker updates a local FAISS index, the API replicas must load the published index version before they can retrieve its contents.

After confirming that the index update is visible to the API, send a question:

curl -X POST \
  http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{"question":"How does RAG use retrieval?"}'

The RAG pipeline should now be able to retrieve information from the uploaded document.

For a reliable integration test, inspect the retrieved document identifiers or source references rather than relying only on the generated answer. An LLM may produce a plausible answer from its existing knowledge even when the new document was not retrieved.

Test Failure Handling

Next, verify that invalid input produces a controlled failure.

Create an empty text file and submit it through the same endpoint. Assuming the text extraction function returns no usable content, our document processor will raise:

ValueError(
    "Document contains no usable text"
)

Because this is a permanent validation error, Celery should not retry the task.

The status endpoint should eventually report:

{
    "task_id": "failed-task-id",
    "status": "FAILURE",
    "error": "Document processing failed"
}

This confirms that the API can report processing failures without exposing internal exception details.

Test Worker Independence

One of the main advantages of background tasks in a RAG API is that API and worker processes operate independently.

Stop the Celery worker:

docker compose stop celery-worker

Submit another document. As long as Redis remains available, FastAPI should still be able to publish the task.

The job will remain queued until a worker becomes available. Because Celery’s result backend does not automatically distinguish every queued task from an unknown task, the status endpoint may continue reporting PENDING.

Restart the worker:

docker compose start celery-worker

The worker should consume the queued job and process it.

This test demonstrates the separation between accepting requests and executing background work. It does not, by itself, prove durability across Redis failures or guarantee recovery from every type of worker interruption.

Measure Background Processing Performance

Finally, compare task processing under different worker configurations.

ConfigurationWhat to Measure
1 worker containerBaseline throughput and task duration
2 worker containersThroughput, CPU, and memory
3 worker containersQueue delay and resource saturation

Use the same document collection for each test and measure both queue waiting time and actual processing time.

Increasing worker count may improve throughput, but it can also increase memory consumption or overload a shared embedding service. The correct configuration depends on the resources used by the ingestion pipeline.

Once these tests pass, we have verified the basic lifecycle of background tasks in a RAG API: document submission, message delivery, worker execution, result tracking, and integration with retrieval.

Next, we will examine production considerations, including queue durability, worker resource limits, task monitoring, and secure document processing.

Production Considerations

Our background processing system now supports document uploads, task execution, retries, status tracking, and independent worker scaling. However, running background tasks in a RAG API in production requires additional safeguards for reliability, security, and resource management.

Protect Redis and Task Data

Redis currently performs three functions: RAG caching, task message delivery, and result storage. Although we use separate logical databases, these workloads still compete for the same server resources.

Redis
  │
  ├── DB 0: RAG Cache
  ├── DB 1: Celery Broker
  └── DB 2: Celery Results

For production workloads, consider separating the Celery broker from the RAG cache. Cache eviction policies that are acceptable for disposable response data may be unsuitable for queued jobs.

Redis persistence, memory limits, authentication, network access, and recovery procedures should be configured according to the reliability requirements of the application.

For critical ingestion workflows, a persistent job database provides an additional record of submitted and completed operations. Redis alone should not be treated as a permanent job history.

Separate Queues by Workload

Not all background jobs consume the same resources. Document extraction may be CPU-intensive, embedding generation may require a GPU, and external API requests may spend most of their time waiting for network responses.

Celery supports dedicated queues for different workloads:

                  Redis Broker
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
     ingestion      embeddings   maintenance
          │            │            │
          ▼            ▼            ▼
     CPU Workers    GPU Workers   CPU Workers

For example, document extraction and embedding generation can eventually run in separate queues with independently configured workers.

This prevents a large batch of document ingestion jobs from consuming resources reserved for other important operations.

Control Worker Resource Consumption

Increasing Celery concurrency does not automatically increase throughput. Every worker process may load embedding models, document parsers, and other application resources.

For memory-intensive tasks, start with low concurrency and monitor resource consumption.

Celery also supports recycling worker processes after a configured number of tasks:

celery_app.conf.update(
    worker_max_tasks_per_child=100
)

This can help limit the impact of memory growth in long-running worker processes, although it does not replace fixing memory leaks or correctly releasing resources.

The optimal setting depends on model initialization costs and the characteristics of the processing workload.

Monitor Queue Length and Task Duration

HTTP metrics alone are insufficient for monitoring background processing.

A Celery system should track:

  • Number of queued tasks.
  • Time spent waiting in the queue.
  • Task execution duration.
  • Successful and failed tasks.
  • Retry frequency.
  • Worker CPU and memory utilization.

Queue waiting time is particularly important. A task may execute quickly but still take several minutes to complete because it spent most of that time waiting for an available worker.

Job Submitted
      │
      ▼
Queue Waiting Time
      │
      ▼
Task Execution Time
      │
      ▼
Job Completed

Measure these stages separately to determine whether additional workers are needed or whether the processing pipeline itself requires optimization.

Secure Document Processing

Uploaded documents should be treated as untrusted input. Restrict file types and sizes, validate actual file contents, and ensure that document parsers receive appropriate resource limits.

Avoid accepting arbitrary filesystem paths from clients. Our API generates its own document identifiers and storage paths, reducing the risk of clients accessing unrelated server files.

For multi-user applications, every job should also be associated with an authenticated user. Status endpoints must verify ownership before returning job information.

Worker containers should run with minimal permissions and access only the storage and services required for their tasks.

Handle Shutdowns and Deployments

Background jobs may continue running while new application versions are deployed. Workers should receive sufficient time to complete active tasks during graceful shutdown.

Long-running jobs require particular attention because container termination, worker crashes, and broker connection failures can interrupt processing.

Production deployments should define explicit policies for task recovery, acknowledgement behavior, retry limits, and duplicate execution.

These policies must be tested rather than assumed. For example, terminate a worker during document processing and verify whether the job is recovered, retried, or marked as failed.

Scale Workers Based on Queue Pressure

The number of API replicas and Celery workers should be controlled independently.

High HTTP Traffic
       │
       ▼
Scale FastAPI Replicas


Growing Task Queue
       │
       ▼
Scale Celery Workers

However, adding workers is useful only when sufficient CPU, GPU, memory, and downstream service capacity remain available.

For example, if embedding generation is limited by an external API rate limit, adding more workers may increase retry frequency without improving throughput.

A better scaling decision considers queue length, waiting time, task duration, resource utilization, and downstream capacity together.

Keep Document Index Updates Consistent

Our ingestion workers operate independently from the FastAPI replicas. Therefore, a successful task must not publish an incomplete or inconsistent vector index.

For local FAISS deployments, a safer approach is to build a new index version, validate it, publish it atomically, and then notify API replicas to reload the updated version.

For larger distributed systems, a dedicated vector database may simplify concurrent indexing and retrieval.

The objective is to ensure that users never query a partially written index and that repeated task execution does not introduce duplicate records.

Start Simple and Add Complexity When Needed

A small production deployment may work well with one Redis instance, a few Celery workers, shared document storage, and a persistent job database.

Separate queues, GPU workers, dedicated embedding services, and distributed vector storage should be introduced when measurements justify the additional complexity.

The central principle behind reliable background tasks in a RAG API is to treat background processing as an independent subsystem with its own capacity, monitoring, security, and recovery requirements.

Where to Go Next

We have extended our production RAG assistant with a distributed task queue that processes documents independently from HTTP requests. Adding background tasks in a RAG API creates new opportunities for automating document ingestion, embedding generation, index maintenance, and other long-running operations.

Improve API Deployment and Scalability

If you are building the infrastructure from the beginning, start with Deploy a RAG Assistant with FastAPI and Dockerize a RAG Assistant with FastAPI.

Our tutorial on How to Scale a RAG API with FastAPI and Docker explains how multiple API replicas, load balancing, and shared Redis improve request-handling capacity. Celery extends this architecture by allowing background workers to scale independently.

Add Monitoring and Distributed Tracing

Background jobs introduce additional execution stages that should be monitored separately from HTTP requests.

In Add Logging and Monitoring to a RAG API, we examine how to collect application metrics and identify performance problems.

For tracking operations across multiple services, see Add Distributed Tracing to a RAG API with OpenTelemetry. The same principles can be extended to Celery workers by propagating trace context between the API and background tasks.

Optimize Redis Usage

Redis now supports both our caching layer and Celery infrastructure. These workloads have different reliability and memory requirements.

Review How to Add Caching to a RAG API with Redis to understand response caching, expiration policies, and cache performance.

Evaluate the Complete RAG Pipeline

Moving document ingestion into background workers changes how documents enter the knowledge base, but it does not automatically improve retrieval quality.

Use Evaluate a RAG System in Python to measure retrieval relevance and answer quality after indexing new documents.

The next logical development is to introduce a more advanced ingestion pipeline with separate processing stages, batch operations, and controlled index updates. This would allow document extraction, chunking, embedding generation, and indexing to scale independently.

Together, these tutorials provide the foundation for operating background tasks in a RAG API as part of a larger production system rather than as isolated asynchronous functions.

Frequently Asked Questions

What are background tasks in a RAG API?

Background tasks in a RAG API are operations executed independently from the HTTP request that initiated them. They are particularly useful for document ingestion, text extraction, embedding generation, vector index updates, and batch processing.

Instead of waiting for an operation to finish, the client receives a task identifier and can check its status later.

Why use Celery instead of FastAPI BackgroundTasks?

FastAPI provides a built-in BackgroundTasks mechanism for executing operations after sending an HTTP response. It is useful for relatively simple tasks that do not require a separate distributed execution system.

Celery provides independent worker processes, message queues, retries, task results, and horizontal worker scaling. It is generally more appropriate for long-running or resource-intensive RAG operations.

Does Celery make RAG responses faster?

Not directly. Celery does not automatically reduce embedding generation, retrieval, or LLM inference time.

Its primary advantage is separating long-running operations from interactive requests. Moving document ingestion into dedicated workers can help preserve API responsiveness while large document collections are processed.

Can Redis be used for both RAG caching and Celery?

Yes. Redis can serve as a response cache, Celery message broker, and Celery result backend.

For smaller deployments, separate Redis logical databases may be sufficient. However, logical databases share the same server resources and eviction configuration. Production systems with stricter reliability requirements may benefit from separate Redis instances.

How do I track Celery task progress?

Celery provides task states such as PENDING, STARTED, RETRY, SUCCESS, and FAILURE.

FastAPI can expose these states through a status endpoint using AsyncResult. For more detailed progress reporting, Celery tasks can publish custom states containing information such as the number of processed documents or completed chunks.

For persistent tracking and user-specific authorization, a separate job database is recommended.

What happens if a Celery worker crashes?

The outcome depends on the task acknowledgement configuration, broker behavior, and type of failure.

With Celery’s default early acknowledgement behavior, an interrupted task may not be automatically executed again. Late acknowledgements can support redelivery in certain failure scenarios, but they also introduce the possibility of duplicate execution.

Document-processing tasks should therefore be designed to handle retries safely.

Can multiple Celery workers update the same FAISS index?

Multiple workers should not independently modify the same FAISS index file without coordination. Concurrent writes can produce inconsistent or corrupted index data.

One approach is to generate immutable index versions and publish updates through a controlled process. Another is to use a dedicated vector database designed for concurrent indexing and retrieval.

The appropriate solution depends on the size of the knowledge base and how frequently documents change.

How many Celery workers should a RAG application use?

There is no universal worker count. The correct configuration depends on CPU availability, GPU memory, model size, task duration, and external service limits.

Start with a small number of workers and measure queue waiting time, task throughput, memory consumption, and processing latency.

Adding workers beyond the capacity of the underlying infrastructure may increase resource contention without improving performance.

Should every RAG request be processed through Celery?

No. Interactive question-answering requests generally benefit from synchronous APIs or streaming responses when users expect immediate results.

Celery is better suited to operations that can complete independently, such as document ingestion, batch evaluation, scheduled index rebuilding, and large-scale embedding generation.

The decision should depend on the expected response time, resource requirements, and whether the client needs an immediate result.

What is the biggest challenge when adding background tasks to a RAG system?

One of the most important challenges is maintaining consistency between background processing and the knowledge base used for retrieval.

A task may successfully generate embeddings but fail while publishing an updated vector index. Retrying that task must not introduce duplicate records or expose incomplete data to users.

Reliable background tasks in a RAG API therefore require idempotent operations, controlled index updates, persistent job tracking, and appropriate failure recovery mechanisms.

Conclusion

Adding background tasks in a RAG API allows us to separate long-running operations from interactive question answering. Instead of processing large documents inside HTTP requests, FastAPI can submit jobs to Redis while independent Celery workers handle document extraction, chunking, embedding generation, and vector index updates.

In this tutorial, we extended our existing RAG assistant with a distributed task queue, implemented document-processing tasks, created endpoints for submitting jobs and checking their status, configured retries, and deployed Celery workers using Docker Compose.

The resulting architecture separates three responsibilities:

FastAPI
   │
   └── Accept Requests
           │
           ▼
       Redis Broker
           │
           └── Queue Jobs
                   │
                   ▼
              Celery Workers
                   │
                   ├── Process Documents
                   ├── Generate Embeddings
                   └── Update Vector Index
                           │
                           ▼
                      Task Results

This separation allows HTTP API replicas and background workers to scale independently. Increasing document-processing capacity no longer requires increasing the number of FastAPI instances.

However, a task queue introduces new engineering responsibilities. Reliable document ingestion requires controlled retries, idempotent operations, shared storage, consistent vector index updates, and monitoring of queue waiting times and worker performance.

The most important principle is to use background processing where it provides a measurable benefit. Interactive RAG requests can remain synchronous, while expensive operations execute independently.

With FastAPI, Celery, and Redis working together, we now have a practical foundation for background tasks in a RAG API that can evolve into a larger distributed document-processing system.

Scroll to Top