Getting Started

Key Concepts

Long-term Memory, Users, Sessions, and more.

Zep is a long-term memory service for AI Assistant applications. With Zep, you can build accurate and personalized Assistant applications by recalling past conversations and relevant business data.

Zep also offers tools to build more determinstic and hallucination-free applications:

  • Dialog Classification is a flexible low-latency API for understanding intent, segmenting users, determining the state of a conversation and more, allowing you to select appropriate prompts and models, and manage application flow.
  • Structured Data Extraction extracts data from conversations with high-fidelity and low-latency, enabling you to confidently populate your data store, call third-party applications, and build custom workflows.

Long-term Memory

Incorporating an entire chat history into a prompt can lead to hallucinations, poor recall, and slow, costly LLM calls. Our goal is to enable you to ground the LLM in appropriate context by incorporating relevant historical data from chat histories, and other user and business data, into your prompts, resulting in significantly improved LLM comprehension and knowledge.

The RAG (Retrieval-Augmented Generation) method for accessing past conversations relevant to current messages presents significant challenges. Merely embedding and subzsequently retrieving messages related to the current dialogue is ineffective. Individual messages often lack sufficient context, and the retrieval is based on relevance rather than chronological order, leading to LLM confusion.

Facts

Zep addresses the challenges presented by RAG by extracting key observations or facts from chat histories and continuously updating a fact table as the conversation evolves. This process ensures that conflicting facts are resolved and deduplicated, and the relationship between conflicting information is preserved. A stylized example below:

FromTo
Robbie only wears Nike shoes.Robbie used to only wear Nike shoes, but she now prefers Adidas.
Robbie’s Nikes fell apart, so she prefers Adidas.Robbie’s Nike shoes fell apart.

Learn more about Facts

Summaries

Zep also generates summaries of the chat history as new messages are added to a session. These summaries are generated progressively, with new messages added to the the prior summary to generate a new summary.

Summaries are lossy. Information from earlier messages is gradually displaced by newer information. Zep supports search over historical summaries in order to retrieve relevant information from prior summaries.

Robbie seeks running shoes under $120, preferably Nike or Adidas. The assistant suggests the Brooks Ghost 14 for pronation issues, but Robbie prefers Nike or Adidas. The assistant recommends the Adidas Ultraboost 21 for comfort, energy return, and stability, on sale for $129.99, which Robbie decides to purchase. The Robbie for the Adidas Ultraboost 21, impressed by its features and price. The assistant requests Robbie’s credit card information to finalize the purchase securely.

Summaries may be used in prompts to augment Facts with more nuanced information.

Learn more about Summaries

Chat History and Sessions

Sessions represent a conversation, and can be associated with Users in a one-to-many relationship.

Chat messages are added to Sessions, with each session having many messages. Facts are extracted from a Session’s chat history, and are stored in the Session’s fact table. Summaries are also generated for the chat history of a specific Session.

Building a Prompt with Zep

Zep’s artifacts, such as Facts, Summaries, and Chat History, can be included in your prompts to augment the LLM’s understanding of the conversation. Typically, you’d use a Chat API and structure your prompts as a series of messages: System, User, Assistant` etc.

More on this in the Building a Prompt section.

MessageTypeContent
SystemYour system prompt. Include Zep artifacts such as relevant {facts} and the most recent {summary} generated prior to the historical dialog below.
AssistantAn assistant message stored in Zep
UserA user message stored in Zep
AssistantAn assistant message stored in Zep
UserA user message stored in Zep
AssistantAn assistant message stored in Zep
UserThe latest user message