AI Semantic Cache

AI License Required

AI Gateway Enterprise: This plugin is only available as part of our AI Gateway Enterprise offering.

The AI Semantic Cache plugin stores user requests to an LLM in a vector database based on semantic meaning. When a similar query is made, it uses these embeddings to retrieve relevant cached requests efficiently.

What is semantic caching?

Semantic caching enhances data retrieval efficiency by focusing on the meaning or context of queries rather than just exact matches. It stores requests based on the underlying intent and semantic similarities between different queries and can then retrieve those cached queries when a similar request is made.

When a new request is made, the system can retrieve and reuse previously cached requests if they are contextually relevant, even if the phrasing is different. This method reduces redundant processing, speeds up response times, and ensures that answers are more relevant to the user’s intent, ultimately improving overall system performance and user experience.

For example, if a user asks, “how to integrate our API with a mobile app” and later asks, “what are the steps for connecting our API to a smartphone application?”, the system understands that both questions are asking for the same information. It can then retrieve and reuse previously cached responses, even if the wording is different. This approach reduces processing time and speeds up responses.

The AI Semantic Cache plugin may not be ideal if the following are true:

You have limited hardware or budget. Storing semantic vectors and running similarity searches require a lot of storage and computing power, which could be an issue.
Your data doesn’t rely on semantics, or exact matches work fine. In this case, semantic caching may offer little benefit. Traditional or keyword-based caching might be more efficient.

How it works

Semantic caching with the AI Semantic Cache plugin involves three parts: request handling, embedding generation, and response caching.

First, a user starts a chat request with the LLM. The AI Semantic Cache plugin queries the vector database to see if there are any semantically similar requests that have already been cached. If there is a match, the vector database returns the cached response to the user.

plaintext

Vector databaseKong Gateway/AI Semantic Cache pluginVector databaseKong Gateway/AI Semantic Cache plugin#mermaid-1781274681223{font-family:"Inter",system-ui,sans-serif;font-size:15px;fill:#495c64;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-1781274681223 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-1781274681223 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-1781274681223 .error-icon{fill:hsl(180, 0%, 100%);}#mermaid-1781274681223 .error-text{fill:rgb(0, 0, 0);stroke:rgb(0, 0, 0);}#mermaid-1781274681223 .edge-thickness-normal{stroke-width:1px;}#mermaid-1781274681223 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1781274681223 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1781274681223 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-1781274681223 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1781274681223 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1781274681223 .marker{fill:#99b0c0;stroke:#99b0c0;}#mermaid-1781274681223 .marker.cross{stroke:#99b0c0;}#mermaid-1781274681223 svg{font-family:"Inter",system-ui,sans-serif;font-size:15px;}#mermaid-1781274681223 p{margin:0;}#mermaid-1781274681223 .actor{stroke:rgb(var(--color-icon-primary));fill:#fff;stroke-width:1;}#mermaid-1781274681223 rect.actor.outer-path[data-look="neo"]{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681223 rect.note[data-look="neo"]{stroke:hsl(52.6829268293, 60%, 73.9215686275%);fill:#fff5ad;filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681223 text.actor>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681223 .actor-line{stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681223 .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-1781274681223 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#495c64;}#mermaid-1781274681223 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#495c64;}#mermaid-1781274681223 [id$="-arrowhead"] path{fill:#495c64;stroke:#495c64;}#mermaid-1781274681223 .sequenceNumber{fill:#fff;}#mermaid-1781274681223 [id$="-sequencenumber"]{fill:#495c64;}#mermaid-1781274681223 [id$="-crosshead"] path{fill:#495c64;stroke:#495c64;}#mermaid-1781274681223 .messageText{fill:#495c64;stroke:none;}#mermaid-1781274681223 .labelBox{stroke:rgb(var(--color-icon-primary));fill:#fff;filter:none;}#mermaid-1781274681223 .labelText,#mermaid-1781274681223 .labelText>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681223 .loopText,#mermaid-1781274681223 .loopText>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681223 .sectionTitle,#mermaid-1781274681223 .sectionTitle>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681223 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:rgb(var(--color-icon-primary));fill:rgb(var(--color-icon-primary));}#mermaid-1781274681223 .note{stroke:hsl(52.6829268293, 60%, 73.9215686275%);fill:#fff5ad;}#mermaid-1781274681223 .noteText,#mermaid-1781274681223 .noteText>tspan{fill:#333;stroke:none;font-weight:normal;}#mermaid-1781274681223 .activation0{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681223 .activation1{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681223 .activation2{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681223 .actorPopupMenu{position:absolute;}#mermaid-1781274681223 .actorPopupMenuPanel{position:absolute;fill:#fff;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-1781274681223 .actor-man circle,#mermaid-1781274681223 line{fill:#fff;stroke-width:2px;}#mermaid-1781274681223 g rect.rect{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681223 .node .neo-node{stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681223 [data-look="neo"].node rect,#mermaid-1781274681223 [data-look="neo"].cluster rect,#mermaid-1781274681223 [data-look="neo"].node polygon{stroke:url(#mermaid-1781274681223-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681223 [data-look="neo"].node path{stroke:url(#mermaid-1781274681223-gradient);stroke-width:1px;}#mermaid-1781274681223 [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681223 [data-look="neo"].node .neo-line path{stroke:rgb(var(--color-icon-primary));filter:none;}#mermaid-1781274681223 [data-look="neo"].node circle{stroke:url(#mermaid-1781274681223-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681223 [data-look="neo"].node circle .state-start{fill:#000000;}#mermaid-1781274681223 [data-look="neo"].icon-shape .icon{fill:url(#mermaid-1781274681223-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681223 [data-look="neo"].icon-shape .icon-neo path{stroke:url(#mermaid-1781274681223-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681223 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}UserLLM chat requestQuery for semantically similar previous requestsIf response, return it or stream it backUser

If there isn’t a match, the AI Semantic Cache plugin prompts the embeddings LLM to generate an embedding for the response.

plaintext

Embeddings LLMKong Gateway/AI Semantic Cache pluginEmbeddings LLMKong Gateway/AI Semantic Cache plugin#mermaid-1781274681523{font-family:"Inter",system-ui,sans-serif;font-size:15px;fill:#495c64;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-1781274681523 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-1781274681523 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-1781274681523 .error-icon{fill:hsl(180, 0%, 100%);}#mermaid-1781274681523 .error-text{fill:rgb(0, 0, 0);stroke:rgb(0, 0, 0);}#mermaid-1781274681523 .edge-thickness-normal{stroke-width:1px;}#mermaid-1781274681523 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1781274681523 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1781274681523 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-1781274681523 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1781274681523 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1781274681523 .marker{fill:#99b0c0;stroke:#99b0c0;}#mermaid-1781274681523 .marker.cross{stroke:#99b0c0;}#mermaid-1781274681523 svg{font-family:"Inter",system-ui,sans-serif;font-size:15px;}#mermaid-1781274681523 p{margin:0;}#mermaid-1781274681523 .actor{stroke:rgb(var(--color-icon-primary));fill:#fff;stroke-width:1;}#mermaid-1781274681523 rect.actor.outer-path[data-look="neo"]{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681523 rect.note[data-look="neo"]{stroke:hsl(52.6829268293, 60%, 73.9215686275%);fill:#fff5ad;filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681523 text.actor>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681523 .actor-line{stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681523 .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-1781274681523 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#495c64;}#mermaid-1781274681523 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#495c64;}#mermaid-1781274681523 [id$="-arrowhead"] path{fill:#495c64;stroke:#495c64;}#mermaid-1781274681523 .sequenceNumber{fill:#fff;}#mermaid-1781274681523 [id$="-sequencenumber"]{fill:#495c64;}#mermaid-1781274681523 [id$="-crosshead"] path{fill:#495c64;stroke:#495c64;}#mermaid-1781274681523 .messageText{fill:#495c64;stroke:none;}#mermaid-1781274681523 .labelBox{stroke:rgb(var(--color-icon-primary));fill:#fff;filter:none;}#mermaid-1781274681523 .labelText,#mermaid-1781274681523 .labelText>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681523 .loopText,#mermaid-1781274681523 .loopText>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681523 .sectionTitle,#mermaid-1781274681523 .sectionTitle>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681523 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:rgb(var(--color-icon-primary));fill:rgb(var(--color-icon-primary));}#mermaid-1781274681523 .note{stroke:hsl(52.6829268293, 60%, 73.9215686275%);fill:#fff5ad;}#mermaid-1781274681523 .noteText,#mermaid-1781274681523 .noteText>tspan{fill:#333;stroke:none;font-weight:normal;}#mermaid-1781274681523 .activation0{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681523 .activation1{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681523 .activation2{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681523 .actorPopupMenu{position:absolute;}#mermaid-1781274681523 .actorPopupMenuPanel{position:absolute;fill:#fff;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-1781274681523 .actor-man circle,#mermaid-1781274681523 line{fill:#fff;stroke-width:2px;}#mermaid-1781274681523 g rect.rect{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681523 .node .neo-node{stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681523 [data-look="neo"].node rect,#mermaid-1781274681523 [data-look="neo"].cluster rect,#mermaid-1781274681523 [data-look="neo"].node polygon{stroke:url(#mermaid-1781274681523-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681523 [data-look="neo"].node path{stroke:url(#mermaid-1781274681523-gradient);stroke-width:1px;}#mermaid-1781274681523 [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681523 [data-look="neo"].node .neo-line path{stroke:rgb(var(--color-icon-primary));filter:none;}#mermaid-1781274681523 [data-look="neo"].node circle{stroke:url(#mermaid-1781274681523-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681523 [data-look="neo"].node circle .state-start{fill:#000000;}#mermaid-1781274681523 [data-look="neo"].icon-shape .icon{fill:url(#mermaid-1781274681523-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681523 [data-look="neo"].icon-shape .icon-neo path{stroke:url(#mermaid-1781274681523-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681523 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}Generate embeddings for \`config.message_countback\` messagesReturn embeddings

The AI Semantic Cache plugin uses a vector database and cache to store responses to requests. The plugin can then retrieve a cached response if a new request matches the semantics of a previous request, or it can tell the vector database to store a new response if there are no matches.

plaintext

Vector databasePrompt/Chat LLMKong Gateway/AI Semantic Cache pluginVector databasePrompt/Chat LLMKong Gateway/AI Semantic Cache plugin#mermaid-1781274681540{font-family:"Inter",system-ui,sans-serif;font-size:15px;fill:#495c64;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-1781274681540 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-1781274681540 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-1781274681540 .error-icon{fill:hsl(180, 0%, 100%);}#mermaid-1781274681540 .error-text{fill:rgb(0, 0, 0);stroke:rgb(0, 0, 0);}#mermaid-1781274681540 .edge-thickness-normal{stroke-width:1px;}#mermaid-1781274681540 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1781274681540 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1781274681540 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-1781274681540 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1781274681540 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1781274681540 .marker{fill:#99b0c0;stroke:#99b0c0;}#mermaid-1781274681540 .marker.cross{stroke:#99b0c0;}#mermaid-1781274681540 svg{font-family:"Inter",system-ui,sans-serif;font-size:15px;}#mermaid-1781274681540 p{margin:0;}#mermaid-1781274681540 .actor{stroke:rgb(var(--color-icon-primary));fill:#fff;stroke-width:1;}#mermaid-1781274681540 rect.actor.outer-path[data-look="neo"]{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681540 rect.note[data-look="neo"]{stroke:hsl(52.6829268293, 60%, 73.9215686275%);fill:#fff5ad;filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681540 text.actor>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681540 .actor-line{stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681540 .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-1781274681540 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#495c64;}#mermaid-1781274681540 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#495c64;}#mermaid-1781274681540 [id$="-arrowhead"] path{fill:#495c64;stroke:#495c64;}#mermaid-1781274681540 .sequenceNumber{fill:#fff;}#mermaid-1781274681540 [id$="-sequencenumber"]{fill:#495c64;}#mermaid-1781274681540 [id$="-crosshead"] path{fill:#495c64;stroke:#495c64;}#mermaid-1781274681540 .messageText{fill:#495c64;stroke:none;}#mermaid-1781274681540 .labelBox{stroke:rgb(var(--color-icon-primary));fill:#fff;filter:none;}#mermaid-1781274681540 .labelText,#mermaid-1781274681540 .labelText>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681540 .loopText,#mermaid-1781274681540 .loopText>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681540 .sectionTitle,#mermaid-1781274681540 .sectionTitle>tspan{fill:#495c64;stroke:none;}#mermaid-1781274681540 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:rgb(var(--color-icon-primary));fill:rgb(var(--color-icon-primary));}#mermaid-1781274681540 .note{stroke:hsl(52.6829268293, 60%, 73.9215686275%);fill:#fff5ad;}#mermaid-1781274681540 .noteText,#mermaid-1781274681540 .noteText>tspan{fill:#333;stroke:none;font-weight:normal;}#mermaid-1781274681540 .activation0{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681540 .activation1{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681540 .activation2{fill:#fff;stroke:#c2d4e0;}#mermaid-1781274681540 .actorPopupMenu{position:absolute;}#mermaid-1781274681540 .actorPopupMenuPanel{position:absolute;fill:#fff;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-1781274681540 .actor-man circle,#mermaid-1781274681540 line{fill:#fff;stroke-width:2px;}#mermaid-1781274681540 g rect.rect{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681540 .node .neo-node{stroke:rgb(var(--color-icon-primary));}#mermaid-1781274681540 [data-look="neo"].node rect,#mermaid-1781274681540 [data-look="neo"].cluster rect,#mermaid-1781274681540 [data-look="neo"].node polygon{stroke:url(#mermaid-1781274681540-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681540 [data-look="neo"].node path{stroke:url(#mermaid-1781274681540-gradient);stroke-width:1px;}#mermaid-1781274681540 [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681540 [data-look="neo"].node .neo-line path{stroke:rgb(var(--color-icon-primary));filter:none;}#mermaid-1781274681540 [data-look="neo"].node circle{stroke:url(#mermaid-1781274681540-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681540 [data-look="neo"].node circle .state-start{fill:#000000;}#mermaid-1781274681540 [data-look="neo"].icon-shape .icon{fill:url(#mermaid-1781274681540-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681540 [data-look="neo"].icon-shape .icon-neo path{stroke:url(#mermaid-1781274681540-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-1781274681540 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}UserMake LLM requestReceive responseStore vectorsStore response message optionsReturn realtime responseUser

Cache management

With the AI Semantic Cache plugin, you can configure a cache of your choice to store the responses from the LLM.

The AI Semantic Cache plugin supports Redis as a cache.

Caching mechanisms

The AI Semantic Cache plugin improves how AI systems provide responses by using two kinds of caching mechanisms:

Exact Caching: This stores precise, unaltered responses for specific queries. If a user asks the same question multiple times, the system can quickly retrieve the pre-stored response rather than generating it again each time. This speeds up response times and reduces computational load.
Semantic Caching: This approach is more flexible and involves storing responses based on the meaning or intent behind the queries. Instead of relying on exact matches, the system can understand and reuse information that is conceptually similar. For instance, if a user asks about “Italian restaurants in New York City” and later about “New York City Italian cuisine,” semantic caching can help provide relevant information based on their related meanings.

Together, these caching methods enhance the efficiency and relevance of AI responses, making interactions faster and more contextually accurate.

When Exact Caching is enabled, the AI Semantic Cache plugin may still return results for queries that are similar but not identical. This is expected behavior: the plugin performs similarity-based caching regardless of the Exact Caching setting.

Headers sent to the client

When the AI Semantic Cache plugin is active, Kong Gateway sends additional headers indicating the cache status and other relevant information:

plaintext

X-Cache-Status: Hit
X-Cache-Status: Miss
X-Cache-Status: Bypass
X-Cache-Status: Refresh
X-Cache-Key: <cache_key>
X-Cache-Ttl: <ttl>
Age: <age>

These headers help clients understand whether a response was served from the cache, if the cache key was used, the remaining time-to-live, and the age of the cached response.

Cache control headers

The plugin respects cache control headers to determine if requests and responses should be cached or not. It supports the following directives:

no-store: Prevents caching of the request or response
no-cache: Forces validation with the origin server before serving the cached response
private: Ensures the response is not cached by shared caches
max-age and s-maxage: Sets the maximum age of the cached response. This causes the vector database to drop and delete the cached response message after expiration, so it’s never seen again.

As most AI services always send no-cache in the response headers, setting cache_control to true will always result in a cache bypass. Only consider setting no-cache if you are using self-hosted services and have control over the response Cache Control headers.

Partials v3.13+

This plugin supports vectordb and embeddings Partials, which let you define shared vector database and embeddings configuration once and reuse it across multiple AI Gateway plugins. This is useful when running this plugin alongside others that use the same vector database and embeddings model, such as AI Semantic Cache, AI RAG Injector, AI Semantic Prompt Guard, AI Semantic Response Guard, and AI Proxy Advanced.

Partial type	Fields covered
`vectordb`	`config.vectordb`
`embeddings`	`config.embeddings`

For setup instructions, see AI plugin Partials.

Vector databases

A vector database can be used to store vector embeddings, or numerical representations, of data items. For example, a response would be converted to a numerical representation and stored in the vector database so that it can compare new requests against the stored vectors to find relevant cached items.

The AI Semantic Cache plugin supports the following vector databases:

Using config.vectordb.strategy: redis and parameters in config.vectordb.redis:
- Redis with Vector Similarity Search (VSS)
  - Redis Cloud
  - Valkey v3.14+: When you configure vectordb.strategy: redis, Kong Gateway queries the server and checks the server name field. If it detects Valkey request, it automatically uses the Valkey-specific driver.
  - Managed Redis with cloud authentication:
  - AWS ElastiCache (auth_provider: aws) - Azure Managed Redis (auth_provider: azure) - Google Cloud Memorystore (auth_provider: gcp) For configuration details, see Using cloud authentication with Redis.
Using config.vectordb.strategy: pgvector and parameters in config.vectordb.pgvector:
- PostgreSQL with pgvector v3.10+

To learn more about vector databases in AI Gateway, see Embedding-based similarity matching in Kong AI gateway plugins.

Using cloud authentication with Redis v3.13+

If your plugin uses a Redis datastore, you can authenticate to it with a cloud Redis provider. This allows you to seamlessly rotate credentials without relying on static passwords.

The following providers are supported:

AWS ElastiCache
Azure Managed Redis
Google Cloud Memorystore (with or without Valkey)

You need:

A running Redis instance on an AWS ElastiCache instance for Valkey 7.2 or later or ElastiCache for Redis OSS version 7.0 or later
The ElastiCache user needs to set “Authentication mode” to “IAM”

The following policy assigned to the IAM user/IAM role that is used to connect to the ElastiCache:

plaintext

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "elasticache:Connect"
            ],
            "Resource": [
                "arn:aws:elasticache:ARN_OF_THE_ELASTICACHE",
                "arn:aws:elasticache:ARN_OF_THE_ELASTICACHE_USER"
            ]
        }
    ]
}

To configure cloud authentication with Redis, add the following parameters to your plugin configuration:

plaintext

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

Replace the following with your actual values:

$INSTANCE_ADDRESS: The ElastiCache instance address.
$INSTANCE_USERNAME: The ElastiCache username with IAM Auth mode configured.
$AWS_CACHE_NAME: Name of your AWS ElastiCache instance.
$AWS_REGION: Your AWS ElastiCache instance region.
$AWS_ACCESS_KEY_ID: (Optional) Your AWS access key ID.
$AWS_ACCESS_SECRET_KEY: (Optional) Your AWS secret access key.