Performance guide

This guide describes application-side methods that reduce latency when you use Zep.

Retrieval latency depends on the graph size, query, filters, result limits, deployment, and application network path. Measure end-to-end latency with representative data and requests. The methods below reduce avoidable application-side latency.

Reuse the Zep SDK Client

The Zep SDK client maintains an HTTP connection pool. Reuse the client to avoid creating a connection for each request:

• Create a single client instance and reuse it across your application
• Avoid creating new client instances for each request or function
• Consider implementing a client singleton pattern in your application
• For serverless environments, initialize the client outside the handler function

Optimizing Context Operations

The thread.add_messages and thread.get_user_context methods are optimized for conversational messages and low-latency retrieval. For optimal performance:

• Use graph.add for larger documents, tool outputs, or business data (up to 10,000 characters per call)
• Chunk large documents before adding them to the graph
• Remove unnecessary metadata or content before persistence
• For bulk document ingestion, use zep-ingest or the Batch API rather than parallel graph.add calls

# Recommended for conversations
zep_client.thread.add_messages(
    thread_id="thread_123",
    messages=[
        {
            "role": "user",
            "name": "Alice",
            "content": "What's the weather like today?"
        }
    ]
)
# Recommended for large documents
zep_client.graph.add(
    data=document_content,  # Your chunked document content
    user_id=user_id,       # Or graph_id
    type="text"            # Can be "text", "message", or "json"
)

Get the Context Block sooner

You can request the Context Block directly in the response to the thread.add_messages() call. This optimization eliminates the need for a separate thread.get_user_context() call. Read more about our Context Block.

In this scenario you can pass in the return_context=True flag to the thread.add_messages() method. Zep will perform a user graph search right after persisting the data and return the context relevant to the recently added messages.

memory_response = await zep_client.thread.add_messages(
    thread_id=thread_id,
    messages=messages,
    return_context=True
)
context = memory_response.context

Searching the Graph Sooner

Use graph search and Advanced Context Block construction when you need custom parameters. You can search the graph while another request adds data.

import asyncio
from zep_cloud.client import AsyncZep
from zep_cloud.types import Message
client = AsyncZep(api_key="your-zep-api-key")
async def add_and_retrieve_from_zep(messages):
    # Concatenate message content to create query string
    query = " ".join([msg.content for msg in messages])
    
    # Execute all operations concurrently
    add_result, edges_result, nodes_result = await asyncio.gather(
        client.thread.add_messages(
            thread_id=thread_id,
            messages=messages
        ),
        client.graph.search(
            user_id=user_id,
            query=query,
            scope="edges"
        ),
        client.graph.search(
            user_id=user_id,
            query=query,
            scope="nodes"
        )
    )
    
    return add_result, edges_result, nodes_result

Construct a custom Context Block from the search results. Advanced Context Block construction provides examples.

Optimizing Search Queries

Zep uses hybrid retrieval combining semantic (vector) similarity, BM25 full-text search, and graph traversal in a single ranked result. For optimal performance:

• Keep your queries concise. The public graph-search API accepts a maximum of 400 characters and rejects a longer query.
• Longer queries may not improve search quality and will increase latency
• Consider breaking down complex searches into smaller, focused queries
• Use specific, contextual queries rather than generic ones

Best practices for search:

• Keep search queries concise and specific
• Structure queries to target relevant information
• Use natural language queries for better semantic matching
• Consider the scope of your search (graphs versus user graphs)

# Recommended - concise query
results = await zep_client.graph.search(
    user_id=user_id,  # Or graph_id
    query="project requirements discussion"
)
# Not accepted - query exceeds the public limit
results = await zep_client.graph.search(
    user_id=user_id,
    query="very long text with multiple paragraphs..."  # Maximum: 400 characters
)

Warming the User Cache

Use user.warm before a likely retrieval to prepare the user’s graph data. For example, call it when a user logs in to your service or opens your application.

# Warm the user's cache when they log in
client.user.warm(user_id=user_id)

Summary

• Reuse Zep SDK client instances to optimize connection management
• Use appropriate methods for different types of content (thread.add_messages for conversations, graph.add for large documents)
• Keep search queries focused and under the token limit for optimal performance
• Warm the user cache when users log in or open your app for faster retrieval