Key Concepts
Long-term Memory, Users, Sessions, and more.
Zep is a long-term memory service for AI Assistant applications. With Zep, you can build accurate and personalized Assistant applications by recalling past conversations and relevant business data.
Zep also offers tools to build more determinstic and hallucination-free applications:
- Dialog Classification is a flexible low-latency API for understanding intent, segmenting users, determining the state of a conversation and more, allowing you to select appropriate prompts and models, and manage application flow.
- Structured Data Extraction extracts data from conversations with high-fidelity and low-latency, enabling you to confidently populate your data store, call third-party applications, and build custom workflows.
Long-term Memory
Incorporating an entire chat history into a prompt can lead to hallucinations, poor recall, and slow, costly LLM calls. Our goal is to enable you to ground the LLM in appropriate context by incorporating relevant historical data from chat histories, and other user and business data, into your prompts, resulting in significantly improved LLM comprehension and knowledge.
The RAG (Retrieval-Augmented Generation) method for accessing past conversations relevant to current messages presents significant challenges. Merely embedding and subzsequently retrieving messages related to the current dialogue is ineffective. Individual messages often lack sufficient context, and the retrieval is based on relevance rather than chronological order, leading to LLM confusion.
Facts
Zep addresses the challenges presented by RAG by extracting key observations or facts from chat histories and continuously updating a fact table as the conversation evolves. This process ensures that conflicting facts are resolved and deduplicated, and the relationship between conflicting information is preserved. A stylized example below:
Learn more about Facts
Summaries
Zep also generates summaries of the chat history as new messages are added to a session. These summaries are generated progressively, with new messages added to the the prior summary to generate a new summary.
Summaries are lossy. Information from earlier messages is gradually displaced by newer information. We do not recommend relying solely on the Memory API’s summary value to ground the LLM in a conversation. Use relevant_facts instead.
Robbie seeks running shoes under $120, preferably Nike or Adidas. The assistant suggests the Brooks Ghost 14 for pronation issues, but Robbie prefers Nike or Adidas. The assistant recommends the Adidas Ultraboost 21 for comfort, energy return, and stability, on sale for $129.99, which Robbie decides to purchase. The Robbie for the Adidas Ultraboost 21, impressed by its features and price. The assistant requests Robbie’s credit card information to finalize the purchase securely.
Summaries may be used in prompts to augment Facts with more nuanced information.
Learn more about Summaries
Chat History and Sessions
Sessions represent a conversation, and can be associated with Users in a one-to-many relationship.
Chat messages are added to Sessions, with each session having many messages. Facts are extracted from a Session’s chat history, and are stored in the Session’s fact table. Summaries are also generated for the chat history of a specific Session.
Building a Prompt with Zep
Zep’s artifacts, such as Facts, Summaries, and Chat History, can be included in your prompts to augment the LLM’s understanding of the conversation. Typically, you’d use a Chat API and structure your prompts as a series of messages: System, User, Assistant` etc.
More on this in the Building a Prompt section.