How to Add Caching to a RAG API with Redis

How to Add Caching to a RAG API with Redis

Table of Contents

What You Will Build

In this tutorial, we will add caching to a RAG API using Redis. We will continue developing the same production RAG assistant from the previous tutorials rather than creating a separate example application.

Our existing system already exposes the RAG pipeline through FastAPI and includes authentication, rate limiting, structured logging, Prometheus metrics, and OpenTelemetry distributed tracing.

Client
  │
  ▼
FastAPI /ask
  │
  ▼
RAG Pipeline
  │
  ├── Hybrid Retrieval
  ├── Reranking
  ├── Context Building
  └── LLM Generation
  │
  ▼
Answer

Every request currently executes this pipeline even when the same question or intermediate computation has already been processed. That can create unnecessary work.

For example, two identical requests may cause the application to generate the same query embedding, search the same document collection, rerank similar candidates, build the same context, and send another request to the LLM.

Question
   │
   ▼
Embedding
   │
   ▼
Retrieval
   │
   ▼
Reranking
   │
   ▼
Context
   │
   ▼
LLM
   │
   ▼
Answer

Some of these operations are computationally expensive. LLM calls may also introduce external API latency and direct usage costs.

Caching allows us to reuse selected results instead of recomputing them for every request.

Where Redis Fits into the RAG Architecture

We will introduce Redis as a dedicated caching layer beside the RAG application:

                 Client
                    │
                    ▼
              FastAPI /ask
                    │
                    ▼
               Cache Check
                /       \
               /         \
          Cache Hit     Cache Miss
             │             │
             │             ▼
             │        RAG Pipeline
             │             │
             │      ┌──────┴──────┐
             │      │             │
             │   Retrieval       LLM
             │      │             │
             │      └──────┬──────┘
             │             │
             │             ▼
             │        Store Result
             │             │
             └──────┬──────┘
                    ▼
                  Answer

                 Redis
              Cache Storage

However, a RAG system has more than one possible caching point. We do not have to cache only the final answer.

Multiple Levels of RAG Caching

During the tutorial, we will examine several possible cache layers:

Question
   │
   ▼
[Embedding Cache]
   │
   ▼
Embedding
   │
   ▼
[Retrieval Cache]
   │
   ▼
Retrieved Documents
   │
   ▼
Reranking
   │
   ▼
Context
   │
   ▼
LLM
   │
   ▼
[Response Cache]
   │
   ▼
Answer

Each layer solves a different problem.

An embedding cache prevents the application from repeatedly generating the same query embedding. A retrieval cache can reuse search results for repeated queries. A response cache can bypass most of the RAG pipeline when an equivalent request has already been answered.

But these optimizations have different correctness risks. If the underlying knowledge base changes, an old retrieval result or final answer may no longer represent the current documents.

For that reason, learning how to add caching to a RAG API is not simply a matter of putting Redis in front of the LLM.

What We Will Implement

We will extend the existing project with a small caching layer:

rag-assistant/
│
├── main.py
├── rag.py
├── retrieval.py
├── reranking.py
├── monitoring.py
├── tracing.py
│
├── cache.py          <-- new
│
├── Dockerfile
├── docker-compose.yml
└── requirements.txt

The new cache.py module will isolate Redis operations from the rest of the RAG logic. The retrieval and generation code should not need to know how Redis connections, serialization, TTLs, or cache keys are implemented.

By the end of the tutorial, the API will support:

  • Redis-backed caching for RAG requests;
  • deterministic cache keys;
  • expiration with TTL;
  • cache invalidation when knowledge changes;
  • optional embedding and retrieval caching;
  • cache hit and miss metrics;
  • safe fallback when Redis is unavailable.

Cache Hits and Cache Misses

Two concepts will appear throughout the implementation.

A cache hit occurs when Redis already contains a valid result:

Request
   │
   ▼
Redis
   │
   ├── HIT
   │
   ▼
Cached Answer

A cache miss means the requested result is not available or has expired:

Request
   │
   ▼
Redis
   │
   ├── MISS
   │
   ▼
RAG Pipeline
   │
   ▼
New Answer
   │
   ▼
Redis

Our goal is not to maximize the cache hit rate at any cost. The goal is to reuse computations only when doing so does not compromise the correctness or freshness of the RAG response.

The Main Challenge: Cache Invalidation

Consider a cached answer created from version 12 of the knowledge base:

Knowledge Base v12
        │
        ▼
     RAG Query
        │
        ▼
  Cached Answer

Later, documents are updated:

Knowledge Base v13
        │
        ▼
     New Query
        │
        ▼
  Old Cached Answer  ← problem

The question may be identical, but the correct answer may have changed.

This makes cache expiration and invalidation central parts of RAG caching architecture. We will therefore design cache keys and TTL policies with knowledge-base changes in mind rather than treating Redis as permanent storage.

By the end, we will have a caching architecture that can reduce repeated computation and LLM calls while preserving the behavior expected from a retrieval-augmented generation system.

Prerequisites

Before we add caching to a RAG API, we need the existing FastAPI application and RAG pipeline from the previous tutorials. We will add Redis to that architecture without changing the core retrieval, reranking, and generation logic.

You should have Python installed and a working RAG API with an /ask endpoint similar to this:

from fastapi import (
    Depends,
    FastAPI,
    Request
)


@app.post(
    "/ask",
    dependencies=[
        Depends(verify_api_key)
    ]
)
@limiter.limit(RAG_RATE_LIMIT)
async def ask(
    request: Request,
    body: AskRequest
):
    request_id = (
        request.state.request_id
    )

    answer = ask_rag(
        body.question,
        request_id=request_id
    )

    return {
        "answer": answer
    }

The endpoint passes the user question to the existing ask_rag() function.

Existing RAG Pipeline

Our simplified RAG pipeline still follows the same sequence used throughout the tutorial series:

def ask_rag(
    question,
    request_id
):
    candidates = hybrid_search(
        question
    )

    reranked = rerank_documents(
        question,
        candidates
    )

    context_documents = (
        reranked[:5]
    )

    context = build_context(
        context_documents
    )

    answer = generate_answer(
        question,
        context
    )

    return answer

At this point, every call executes the complete pipeline:

Question
   │
   ▼
Hybrid Search
   │
   ▼
Reranking
   │
   ▼
Context Building
   │
   ▼
LLM Generation
   │
   ▼
Answer

We will preserve this behavior for cache misses. Redis will provide an alternative fast path when a valid cached result already exists.

Install the Redis Python Client

To add caching to a RAG API, the Python application needs a Redis client. Install the official Redis Python package:

pip install redis

Add it to requirements.txt as well:

fastapi
uvicorn
slowapi
prometheus-client

opentelemetry-api
opentelemetry-sdk
opentelemetry-instrumentation-fastapi
opentelemetry-exporter-otlp-proto-grpc

redis

openai
sentence-transformers
faiss-cpu
rank-bm25
numpy

Redis itself runs as a separate service. The Python package only provides the client used by the RAG API to communicate with that service.

RAG API
   │
   │ Redis client
   ▼
Redis Server

Add the Cache Module

Create a new file:

cache.py

Our project now looks like:

rag-assistant/
│
├── main.py
├── rag.py
├── retrieval.py
├── reranking.py
├── monitoring.py
├── tracing.py
├── cache.py
│
├── Dockerfile
├── docker-compose.yml
└── requirements.txt

We will keep Redis-specific code inside cache.py. This separation is important because caching is infrastructure, not part of the retrieval algorithm itself.

The RAG pipeline should ask the cache for a value without needing to know how the Redis connection or serialization works.

rag.py
  │
  ├── get cached result
  │
  ├── run RAG if necessary
  │
  └── store new result
        │
        ▼
     cache.py
        │
        ▼
      Redis

Redis Configuration

We will configure the Redis connection through an environment variable rather than hard-coding the server address:

REDIS_URL=redis://localhost:6379/0

When the API and Redis run as Docker services, the hostname will normally be the Redis service name:

REDIS_URL=redis://redis:6379/0

This lets the same application code work in local development and containerized environments.

What We Will Not Change

Adding Redis should not require us to rewrite the RAG system. Hybrid search, FAISS vector retrieval, BM25 retrieval, reranking, context construction, and LLM generation remain independent components.

Instead, RAG API caching becomes another layer around selected expensive operations:

              Cache Layer
                  │
        ┌─────────┴─────────┐
        │                   │
       HIT                 MISS
        │                   │
        ▼                   ▼
Cached Result         Existing RAG Code

This distinction will become important later when we cache different stages independently. A response cache can skip almost the entire pipeline, while an embedding cache only avoids repeated embedding generation.

With the prerequisites ready, we can now add caching to a RAG API at the infrastructure level by connecting Redis to our FastAPI application and Docker environment.

Step 1 — Add Redis to the RAG Architecture

The first implementation step is to connect Redis to the application and create a small reusable cache interface. Before we add caching to a RAG API at the response or retrieval level, the rest of the application needs a reliable way to read and write cached data.

Redis will run independently from FastAPI:

Client
  │
  ▼
FastAPI
  │
  ├──────────────► Redis
  │
  ▼
RAG Pipeline

This separation is useful because Redis can be restarted, scaled, or configured independently from the RAG application.

Start Redis with Docker Compose

Since our RAG assistant is already containerized, add Redis to docker-compose.yml:

services:

  rag-api:
    build: .
    container_name: rag-api

    environment:
      REDIS_URL: redis://redis:6379/0

    depends_on:
      - redis

    ports:
      - "8000:8000"

  redis:
    image: redis:7-alpine
    container_name: rag-redis

    restart: unless-stopped

Inside the Docker network, redis is the service hostname. FastAPI therefore connects to:

redis://redis:6379/0

Port 6379 does not need to be published to the host unless you specifically want to access Redis from outside the Docker network.

Start the services:

docker compose up -d --build

The architecture now becomes:

Docker Network
│
├── rag-api
│      │
│      └── redis://redis:6379
│
└── redis

Create the Redis Client

For connection options, authentication, and client configuration, see the official Redis Python client documentation.

Now open cache.py and create the Redis connection:

import os

import redis


REDIS_URL = os.getenv(
    "REDIS_URL",
    "redis://localhost:6379/0"
)


redis_client = redis.Redis.from_url(
    REDIS_URL,
    decode_responses=True
)

Setting decode_responses=True means Redis responses are returned as Python strings instead of raw bytes. This makes the cache interface easier to use for JSON data.

Create Basic Cache Functions

We do not want the RAG pipeline to call Redis commands directly throughout the application. Instead, create a small abstraction in cache.py:

import json


def get_cache(
    key
):
    value = redis_client.get(
        key
    )

    if value is None:
        return None

    return json.loads(
        value
    )


def set_cache(
    key,
    value,
    ttl=300
):
    redis_client.setex(
        key,
        ttl,
        json.dumps(value)
    )

The rest of the application now interacts with two simple operations:

get_cache(key)

set_cache(
    key,
    value,
    ttl
)

The default TTL is currently 300 seconds, or five minutes. We will design a more deliberate expiration strategy later in the tutorial.

Test the Redis Connection

Before integrating Redis into the RAG pipeline, verify that the application can communicate with it.

redis_client.set(
    "rag:test",
    "connected"
)

value = redis_client.get(
    "rag:test"
)

print(value)

The result should be:

connected

You can also verify the Redis container:

docker exec \
  -it rag-redis \
  redis-cli ping

Redis should respond:

PONG

Use Namespaced Cache Keys

Even at this early stage, we should avoid generic Redis keys such as:

question
answer
result

A better approach is to namespace keys according to their purpose:

rag:response:...
rag:retrieval:...
rag:embedding:...

This becomes important once we add caching to a RAG API at multiple stages. Redis may eventually contain response caches, embeddings, retrieval results, rate-limiting data, and other application state.

Namespacing makes the role of each key explicit:

rag
│
├── response
│
├── retrieval
│
└── embedding

Do Not Make Redis a Hard Dependency

There is one architectural issue with our first implementation. If Redis becomes unavailable, redis_client.get() can raise an exception. We do not want a cache outage to make the entire RAG API unavailable.

Caching is an optimization. The underlying RAG pipeline should still work without it.

Redis available
      │
      ▼
Use Cache


Redis unavailable
      │
      ▼
Run RAG Normally

We can therefore make the cache functions fail gracefully:

def get_cache(
    key
):
    try:
        value = redis_client.get(
            key
        )

        if value is None:
            return None

        return json.loads(
            value
        )

    except redis.RedisError:
        return None


def set_cache(
    key,
    value,
    ttl=300
):
    try:
        redis_client.setex(
            key,
            ttl,
            json.dumps(value)
        )

    except redis.RedisError:
        pass

A Redis failure now behaves like a cache miss. The request continues through the normal RAG pipeline instead of failing because an optimization layer is unavailable.

Request
   │
   ▼
Redis Check
   │
   ├── HIT ──────────► Cached Result
   │
   ├── MISS ─────────► RAG Pipeline
   │
   └── ERROR ────────► RAG Pipeline

In production, we would also log and measure Redis failures rather than silently ignoring them. We will connect caching to our existing observability system later.

At this point, the infrastructure required to add caching to a RAG API is ready. Redis is part of the Docker architecture, the application has a reusable cache interface, and a cache failure does not prevent the RAG pipeline from answering requests.

Next, we can use this infrastructure to implement the first real optimization: caching complete RAG responses so repeated requests can bypass expensive retrieval and LLM generation.

Step 2 — Cache RAG Responses

Now that Redis is connected, we can add caching to a RAG API at the highest level: the final response. If the same request has already produced an answer and the cached value is still valid, the API can return it without executing retrieval, reranking, context construction, or LLM generation again.

This is potentially the most valuable cache because a single cache hit can bypass almost the entire RAG pipeline.

Request
   │
   ▼
Response Cache
   │
   ├── HIT ─────────────► Answer
   │
   └── MISS
          │
          ▼
     Hybrid Search
          │
          ▼
      Reranking
          │
          ▼
       Context
          │
          ▼
         LLM
          │
          ▼
     Store Answer
          │
          ▼
        Answer

Create a Response Cache Key

Redis requires a string key. We could use the question directly:

rag:response:What is hybrid search?

But storing arbitrary user input inside Redis keys is not a good design. Questions may be long, contain private information, or include characters that make keys difficult to inspect and manage.

Instead, we can hash the question:

import hashlib


def create_response_cache_key(
    question
):
    normalized = (
        question
        .strip()
        .lower()
    )

    digest = hashlib.sha256(
        normalized.encode(
            "utf-8"
        )
    ).hexdigest()

    return (
        f"rag:response:{digest}"
    )

The original question:

What is hybrid search?

becomes a key similar to:

rag:response:
9db1e4f86a0c...

Redis no longer needs the complete question inside the key.

Why Normalize the Question?

Without normalization, these requests would produce different hashes:

What is hybrid search?

what is hybrid search?

  What is hybrid search?  

Lowercasing and trimming whitespace makes these simple variations share the same response cache entry.

However, we should not perform aggressive normalization yet. These questions may look similar but have different meanings:

What is NOT supported?

What is supported?

Cache normalization should therefore be deterministic and conservative.

Add Response Caching to the RAG Pipeline

Import the cache functions into rag.py:

from cache import (
    get_cache,
    set_cache
)

Then add the response cache around the existing pipeline:

def ask_rag(
    question,
    request_id
):
    cache_key = (
        create_response_cache_key(
            question
        )
    )

    cached = get_cache(
        cache_key
    )

    if cached is not None:
        return cached["answer"]

    candidates = hybrid_search(
        question
    )

    reranked = rerank_documents(
        question,
        candidates
    )

    context_documents = (
        reranked[:5]
    )

    context = build_context(
        context_documents
    )

    answer = generate_answer(
        question,
        context
    )

    set_cache(
        cache_key,
        {
            "answer": answer
        },
        ttl=300
    )

    return answer

The behavior is now straightforward.

get_cache()
    │
    ├── value exists
    │       │
    │       ▼
    │   return answer
    │
    └── None
            │
            ▼
       execute RAG
            │
            ▼
       set_cache()
            │
            ▼
       return answer

This is the first working implementation to add caching to a RAG API. Repeated requests can now avoid expensive RAG operations entirely.

Test a Cache Miss

Send a question that has not previously been cached:

curl -X POST \
  http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{
    "question":
      "What is hybrid search?"
  }'

The first request follows the complete pipeline:

Redis
  │
  ▼
MISS
  │
  ▼
Hybrid Retrieval
  │
  ▼
Reranking
  │
  ▼
Context
  │
  ▼
LLM
  │
  ▼
Redis SET
  │
  ▼
Answer

Test a Cache Hit

Send exactly the same question again:

curl -X POST \
  http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{
    "question":
      "What is hybrid search?"
  }'

This time the request can follow a much shorter path:

Redis
  │
  ▼
HIT
  │
  ▼
Answer

No embedding generation, vector search, BM25 search, reranking, context construction, or LLM request is required.

The Performance Difference

Suppose an uncached RAG request takes:

Embedding          40 ms
Retrieval          90 ms
Reranking         210 ms
Context             8 ms
LLM              2400 ms
------------------------
Total            2748 ms

A response-cache hit might require only the Redis lookup plus normal API processing:

Redis lookup        few ms
RAG pipeline        skipped
LLM call            skipped

The exact improvement depends on infrastructure and network latency, but response caching can remove the most expensive parts of repeated requests.

Response Caching Can Also Reduce LLM Costs

Latency is not the only reason to cache RAG responses. If the LLM provider charges according to input and output token usage, repeated identical requests create repeated costs.

Without Cache

Question
  │
  ▼
LLM Call ──► Token Cost

Question
  │
  ▼
LLM Call ──► Token Cost

Question
  │
  ▼
LLM Call ──► Token Cost

With a valid response cache:

First Request
  │
  ▼
LLM Call ──► Token Cost
  │
  ▼
Redis


Repeated Request
  │
  ▼
Redis HIT ──► No LLM Call

For applications with repeated questions, this can make RAG API caching useful for both response time and operating cost.

Do Not Assume Similar Questions Are Equivalent

Our current cache performs exact matching after basic normalization. It does not consider these questions equivalent:

What is RAG?

Explain retrieval-augmented generation.

How does a RAG system work?

A semantic cache could use embeddings to find previously answered questions with similar meaning, but that introduces an important correctness problem: semantic similarity does not guarantee that two questions should receive the same answer.

For this tutorial, we will keep response caching deterministic. The cache will reuse a response only when the cache key identifies the same normalized request.

A More Important Problem Remains

Our current key depends only on the question:

cache_key =
    hash(question)

That is not sufficient for a production RAG system.

Imagine that the documents are updated after an answer has been cached:

Question
   │
   ▼
Knowledge Base v1
   │
   ▼
Answer A
   │
   ▼
Redis


Knowledge Base updated to v2
   │
   ▼
Same Question
   │
   ▼
Redis HIT
   │
   ▼
Old Answer A

The cached answer may now be stale even though the question is identical.

Other changes can have the same effect. A different LLM model, retrieval configuration, metadata filter, prompt version, or user scope may require a different response.

Therefore, to safely add caching to a RAG API, the cache key must represent more than the question itself. In the next step, we will design cache keys that include the state and configuration that can change the resulting RAG answer.

Step 3 — Design Reliable Cache Keys

Our current response cache uses only the normalized question. That works for a simple demonstration, but to safely add caching to a RAG API, the key should represent the inputs that can change the final answer.

Consider the same question executed against two versions of the knowledge base:

Question + Knowledge Base v1 → Answer A

Question + Knowledge Base v2 → Answer B

If both requests use the same cache key, the second request may incorrectly receive Answer A.

Include the Knowledge Base Version

A simple solution is to maintain a version for the indexed knowledge:

KNOWLEDGE_VERSION = "13"

Then include it when creating the response cache key:

import hashlib


def create_response_cache_key(
    question,
    knowledge_version
):
    normalized = (
        question
        .strip()
        .lower()
    )

    source = (
        f"{knowledge_version}:"
        f"{normalized}"
    )

    digest = hashlib.sha256(
        source.encode("utf-8")
    ).hexdigest()

    return (
        f"rag:response:{digest}"
    )

Now the same question generates different cache entries when the knowledge base changes:

v12 + question
      │
      ▼
rag:response:abc...


v13 + question
      │
      ▼
rag:response:xyz...

Old entries can remain in Redis until their TTL expires, but they will no longer be returned for requests using the new knowledge version.

Include Other Answer-Changing Inputs

The knowledge base is not the only input that can affect a RAG response. Depending on the application, the key may also need to represent:

  • metadata filters;
  • prompt version;
  • LLM model;
  • retrieval configuration;
  • tenant or user scope when data access differs.

For example:

source = (
    f"{knowledge_version}:"
    f"{prompt_version}:"
    f"{model_name}:"
    f"{normalized_question}"
)

The rule is simple: if changing a parameter can legitimately change the answer, consider whether that parameter belongs in the cache identity.

Do Not Add Everything to the Key

The opposite mistake is including values that do not affect the result:

request_id
timestamp
trace_id

These values are different for every request. Including them would create a new key every time and effectively eliminate response caching.

Good cache identity:

question
+ knowledge version
+ answer-changing configuration


Bad cache identity:

question
+ request ID
+ timestamp

Designing reliable keys is one of the most important parts when you add caching to a RAG API. Redis can return cached data extremely quickly, but it cannot determine whether two requests should logically share the same result.

With cache identity defined, the next problem is deciding how long cached results should remain valid and how they should be invalidated when the RAG system changes.

Step 4 — Add TTL and Cache Invalidation

Reliable cache keys prevent incompatible requests from sharing results, but cached data should not live forever. When we add caching to a RAG API, we also need a strategy for expiration and invalidation.

Set a TTL for Cached Responses

Our set_cache() function already supports a TTL:

set_cache(
    cache_key,
    {
        "answer": answer
    },
    ttl=300
)

Redis automatically deletes this entry after 300 seconds. A later request becomes a cache miss and runs the RAG pipeline again.

Cache Entry
    │
    ├── 0–300 sec → HIT
    │
    └── after TTL → MISS → RAG Pipeline

There is no universal TTL for RAG. Frequently updated knowledge may require short expiration times, while relatively static documentation can tolerate longer caching.

Invalidate the Cache When Knowledge Changes

TTL alone can still return stale answers between a document update and cache expiration. The knowledge version introduced in the previous step provides a cleaner solution.

KNOWLEDGE_VERSION=13

After rebuilding the document index, increment the version:

KNOWLEDGE_VERSION=14

New requests immediately generate different cache keys:

v13 + question → old cache entry

v14 + question → new cache entry

This avoids having to search Redis for every response associated with the previous document set. Old entries simply expire according to their TTL.

Use Explicit Deletion When Necessary

Some applications also need immediate deletion of a specific cache entry. Add a small helper:

def delete_cache(
    key
):
    try:
        redis_client.delete(
            key
        )

    except redis.RedisError:
        pass

This is useful when the application knows exactly which cached result became invalid. For large knowledge-base updates, versioning is usually easier than deleting thousands of individual keys.

Combine Versioning and TTL

A practical strategy is to use both mechanisms:

Knowledge Version
      │
      ├── prevents reuse after updates
      │
      ▼
Cache Key

TTL
      │
      └── removes old entries automatically

Versioning protects correctness, while TTL limits how long unused data occupies Redis memory.

This distinction matters when you add caching to a RAG API: expiration is primarily a storage and freshness policy, while invalidation determines whether a cached result is still logically valid for the current RAG system.

Now that response caching has reliable keys and an invalidation strategy, we can move deeper into the pipeline and cache smaller reusable computations such as embeddings and retrieval results.

Step 5 — Cache Embeddings and Retrieval Results

Response caching provides the largest shortcut, but it is not always appropriate. Sometimes we still want to run reranking and LLM generation while avoiding repeated work earlier in the pipeline. To add caching to a RAG API more selectively, we can cache query embeddings and retrieval results.

Cache Query Embeddings

If the same query appears repeatedly, there is little value in generating its embedding every time.

Question
   │
   ▼
Embedding Cache
   │
   ├── HIT ──► Existing Vector
   │
   └── MISS ─► Embedding Model
                    │
                    ▼
                 Redis

We can use the same hashing approach for the key:

def create_embedding_key(
    question,
    model_name
):
    source = (
        f"{model_name}:"
        f"{question.strip().lower()}"
    )

    digest = hashlib.sha256(
        source.encode("utf-8")
    ).hexdigest()

    return (
        f"rag:embedding:{digest}"
    )

Including the embedding model is important because changing models can produce a different vector for the same text.

The cached value can then be used before calling the embedding model:

key = create_embedding_key(
    question,
    embedding_model_name
)

embedding = get_cache(
    key
)

if embedding is None:
    embedding = create_embedding(
        question
    )

    set_cache(
        key,
        embedding,
        ttl=3600
    )

Cache Retrieval Results

We can apply the same pattern around hybrid search:

Question
   │
   ▼
Retrieval Cache
   │
   ├── HIT ──► Candidates
   │
   └── MISS
          │
          ▼
   BM25 + Vector Search
          │
          ▼
       Candidates
          │
          ▼
        Redis

A retrieval cache key should include the knowledge version because document updates can change the search results.

source = (
    f"{knowledge_version}:"
    f"{retrieval_config}:"
    f"{normalized_question}"
)

Metadata filters must also be included when they affect which documents can be retrieved. Otherwise, one request could accidentally reuse results generated for a different filter or data scope.

Choose the Cache Layer Deliberately

Different cache levels provide different shortcuts:

Response Cache
   └── skips almost entire RAG pipeline

Retrieval Cache
   └── skips search

Embedding Cache
   └── skips embedding generation

This does not mean that every production system should enable all three. Each additional cache creates more state, invalidation rules, and operational complexity.

A good approach when you add caching to a RAG API is to start with the expensive repeated operation you can safely reuse, measure its effect, and introduce additional cache layers only when they provide a clear benefit.

Next, we will measure those benefits directly by tracking cache hits, misses, and their effect on RAG API performance.

Step 6 — Measure Cache Performance

Caching should be measured rather than assumed to be useful. After we add caching to a RAG API, we need to know how often Redis actually prevents expensive RAG operations.

Track Cache Hits and Misses

Since our API already uses Prometheus, add two counters:

from prometheus_client import (
    Counter
)


cache_hits = Counter(
    "rag_cache_hits_total",
    "Number of RAG cache hits",
    ["cache_type"]
)

cache_misses = Counter(
    "rag_cache_misses_total",
    "Number of RAG cache misses",
    ["cache_type"]
)

Then record the result of each cache lookup:

cached = get_cache(
    cache_key
)

if cached is not None:
    cache_hits.labels(
        cache_type="response"
    ).inc()

    return cached["answer"]

cache_misses.labels(
    cache_type="response"
).inc()

The same metric can distinguish our cache layers:

cache_type="response"
cache_type="retrieval"
cache_type="embedding"

Calculate the Cache Hit Rate

The most useful high-level metric is the cache hit rate:

hit rate =
hits / (hits + misses)

For example:

Cache hits:    720
Cache misses:  280

Hit rate:      72%

A high hit rate can indicate that caching is eliminating substantial repeated work. A low hit rate is not automatically a problem: it may simply mean that most questions are unique.

Compare Request Latency

Cache effectiveness should also appear in the latency metrics and OpenTelemetry traces created in the previous tutorials.

Cache MISS

retrieval      90 ms
reranking     210 ms
context        10 ms
LLM          2500 ms


Cache HIT

Redis lookup    few ms
RAG stages      skipped

For response caching, another useful metric is the number of LLM calls avoided. This connects caching directly to both latency and token costs.

Measure Before Adding More Cache Layers

When you add caching to a RAG API, more caching is not automatically better. If embedding generation represents only a tiny fraction of total latency, an embedding cache may add complexity without producing a meaningful improvement.

Use cache hit rate, request latency, Redis errors, and avoided expensive operations to decide whether each cache layer is worthwhile.

With caching now measurable, we can examine an equally important question: which parts of a RAG system should actually be cached, and which data is better recomputed for every request.

What Should and Should Not Be Cached in RAG

The technical ability to cache a value does not mean that caching it is useful. When we add caching to a RAG API, each cache layer should be evaluated according to cost, reuse frequency, freshness, and correctness.

Good Candidates for Caching

DataWhy Cache It?Main Risk
Query embeddingsAvoid repeated embedding generationEmbedding model changes
Retrieval resultsAvoid repeated vector and BM25 searchKnowledge base changes
Final responsesAvoid most of the RAG pipeline and LLM callsStale or incorrectly shared answers

Query embeddings are relatively straightforward because identical text processed by the same embedding model can usually reuse the same vector. Retrieval results require stronger invalidation because they depend on the current document index.

Final responses offer the largest potential performance improvement, but they also require the most careful cache identity because the answer may depend on documents, filters, prompts, models, and access scope.

Be Careful with User-Specific Data

Suppose two users ask the same question but have access to different documents:

User A ──► Private Documents A
User B ──► Private Documents B

A cache based only on the question could return User A’s result to User B. In systems with tenant-specific or permission-controlled retrieval, authorization scope must therefore be reflected in the cache design.

Do Not Cache Highly Dynamic Results Blindly

Caching is less useful when the underlying information changes rapidly or every request depends on unique context.

Good reuse
Same input + stable knowledge
        │
        ▼
      Cache


Poor reuse
Unique input + changing knowledge
        │
        ▼
    Recompute

The same applies when an answer intentionally depends on current time, session state, conversation history, or frequently changing external data.

The objective when you add caching to a RAG API is therefore not to cache as much as possible. Cache computations that are expensive, repeated, and safe to reuse. Recompute results when freshness or request-specific context is more important than the performance saved by Redis.

Production Caching Considerations

A local Redis cache is easy to demonstrate, but production introduces failure handling, memory limits, security, and concurrency. These issues should be considered before you add caching to a RAG API under real traffic.

Keep Redis Optional

The RAG pipeline should continue working when Redis is temporarily unavailable:

Redis available
   │
   ├── HIT  → cached result
   └── MISS → RAG pipeline

Redis unavailable
   │
   └────────→ RAG pipeline

Cache errors should be logged and measured, but they should not normally turn a valid RAG request into an API error.

Prevent a Cache Stampede

Imagine that a popular cache entry expires while hundreds of clients request it simultaneously. Every request sees a miss and starts the same expensive RAG pipeline:

100 requests
     │
     ▼
Cache expires
     │
     ▼
100 cache misses
     │
     ▼
100 LLM calls

This is known as a cache stampede. At larger scale, distributed locks or request coalescing can allow one request to regenerate the value while others wait for the result.

Set Redis Memory Policies Deliberately

Redis stores cached data in memory, so cache growth must be controlled. TTLs remove expired entries, while Redis memory limits and eviction policies determine what happens when available cache memory is exhausted.

Monitor memory usage together with hit rate. A large cache with almost no hits may consume resources without providing meaningful performance improvements.

Protect Redis

Redis should not be exposed publicly without a specific reason. In our Docker architecture, the API can access Redis through the internal network without publishing port 6379 to the host.

Internet
   │
   ▼
RAG API
   │
   ▼
Private Network
   │
   ▼
Redis

Production deployments should also use the authentication, encryption, and network controls appropriate for their Redis environment.

Watch Serialization Size

Caching a small answer is different from storing hundreds of retrieved documents or large embedding arrays. Large cache values increase memory consumption and network transfer.

Before caching an intermediate RAG result, compare the cost of recomputing it with the cost of storing and transferring it.

Version the Cache with the Application

Knowledge is not the only thing that changes. A new retrieval algorithm, prompt, embedding model, or serialization format may make existing cache entries incompatible.

A simple application-level version can become part of the key:

rag:v2:response:...
rag:v2:retrieval:...
rag:v2:embedding:...

Deploying an incompatible change can then move the application to v3 without trying to migrate every old cache entry.

Optimize from Measurements

The production strategy should ultimately be driven by observability:

Cache hit rate
Redis latency
Redis errors
Memory usage
RAG latency
LLM calls avoided
        │
        ▼
Caching decisions

When you add caching to a RAG API, Redis should reduce expensive repeated work without becoming another fragile dependency. Start with the cache layer that produces a measurable benefit, monitor it, and add complexity only when production data justifies it.

Where to Go Next

We now have another production layer around the RAG assistant. After learning how to add caching to a RAG API, repeated requests no longer have to execute every expensive operation in the pipeline.

Caching works best together with the other components developed throughout this series:

The complete production architecture is gradually becoming:

Client
  │
  ▼
FastAPI
  │
  ├── Authentication
  ├── Rate Limiting
  ├── Redis Cache
  │
  ▼
RAG Pipeline
  │
  ├── Retrieval
  ├── Reranking
  └── LLM
  │
  ├── Logs
  ├── Metrics
  └── Traces

The important principle is that caching remains an optimization rather than part of RAG correctness. When you add caching to a RAG API, the underlying retrieval and generation pipeline should remain capable of producing a fresh answer whenever the cache is missing, invalid, expired, or unavailable.

Frequently Asked Questions

Why should I add caching to a RAG API?

Caching can reduce response latency, repeated retrieval work, embedding generation, and unnecessary LLM calls. It is most useful when the RAG application receives repeated or identical requests.

What is the best place to cache a RAG pipeline?

There is no single best layer. Response caching provides the largest shortcut because it can bypass almost the entire pipeline. Retrieval caching avoids repeated searches, while embedding caching prevents repeated vector generation.

Response Cache  → largest shortcut
Retrieval Cache → skips search
Embedding Cache → skips embedding generation

The best choice depends on which operations are expensive and how often their results can safely be reused.

Why use Redis for RAG caching?

Redis provides fast key-value access, TTL-based expiration, atomic operations, and shared storage that can be accessed by multiple API instances. This makes it suitable for production environments where an in-process Python dictionary would not be shared between application replicas.

How long should RAG responses stay in Redis?

The appropriate TTL depends on how quickly the underlying information changes. Static documentation may tolerate longer TTLs, while frequently updated knowledge may require much shorter expiration periods.

TTL should also be combined with explicit cache invalidation or knowledge-base versioning when freshness is important.

What happens when the RAG knowledge base changes?

Cached retrieval results and answers may become stale. One practical solution is to include a knowledge-base version in the cache key. After rebuilding the index, increment the version so new requests no longer use entries created from the previous knowledge base.

Can I cache query embeddings?

Yes. Identical normalized text processed by the same embedding model can often reuse the same query vector. Include the embedding model or its version in the cache key so vectors from incompatible models are not mixed.

Should I use semantic caching for RAG questions?

Semantic caching can reuse answers for questions that are similar rather than identical, but it introduces additional correctness risks. Two questions can have similar embeddings while still requiring different answers.

Exact deterministic caching is therefore a safer starting point when you add caching to a RAG API. Semantic caching should be introduced only when its similarity rules and failure cases can be evaluated properly.

Can cached RAG responses create security problems?

Yes. If users or tenants have access to different documents, a shared cache key based only on the question could expose a response generated from another user’s data. Authorization scope must be part of the cache design whenever retrieval permissions differ.

What happens if Redis goes down?

Caching should normally fail open: the cache lookup fails, but the request continues through the standard RAG pipeline. Redis failures should still be logged and monitored so operational problems are visible.

How do I know whether RAG caching is useful?

Measure cache hits, misses, hit rate, Redis latency, overall request latency, and expensive operations avoided. For response caching, tracking avoided LLM calls can also show the effect on token costs.

The goal when you add caching to a RAG API is not the highest possible hit rate. The goal is to eliminate repeated work while preserving fresh, correct, and properly authorized RAG responses.

Conclusion

Caching can remove a significant amount of repeated work from a production RAG system. In this tutorial, we learned how to add caching to a RAG API with Redis without making the cache responsible for the correctness of the underlying RAG pipeline.

We started with response caching, then designed deterministic cache keys that account for knowledge-base and configuration changes. We added TTL-based expiration and version-based invalidation, and extended the same approach to query embeddings and retrieval results.

Question
   │
   ▼
Response Cache
   │
   ├── HIT ─────────────► Answer
   │
   └── MISS
          │
          ▼
   Embedding Cache
          │
          ▼
   Retrieval Cache
          │
          ▼
      Reranking
          │
          ▼
         LLM
          │
          ▼
     Cache Response

We also connected caching to the observability architecture by measuring hits and misses, and examined production concerns such as Redis failures, cache stampedes, memory usage, security, serialization size, and application versioning.

The most important principle is that caching should remain an optimization. If Redis is unavailable or a cached value is no longer valid, the RAG pipeline must still be able to retrieve current documents and generate a fresh answer.

When you add caching to a RAG API with that principle in mind, Redis can reduce latency and repeated LLM calls while keeping retrieval freshness and answer correctness under control.

Scroll to Top