Structured Outputs from Messages

Many business and consumer apps need to extract structured data from conversation between an Assistant and human user. Often, the extracted data is the objective of the conversation.

Often, you will want to identify the data values you have collected and which values you still need to collect in order to prompt the LLM to request the latter.

This can be a slow and inaccurate exercise, and frustrating to your users. If you’re making multiple calls to an LLM to extract and validate data on every chat turn, you’re likely adding seconds to your response time.

Quick Start

An end-to-end SDE example (in Python) can be found in the Zep By Example repo.

The example covers:

• defining a model using many of the field types that SDE supports
• extracting data from a Chat History
• and provides an example of how to merge newly extracted data with an already partially populated model.

SDE vs JSON or Structured Outputs Mode

Many model providers offer a JSON and/or Structured Outputs inference mode that guarantees the output will be well-formed JSON, or in the case of Structured Output, is valid according to a provided schema.

However:

1. When using JSON Mode, there are no guarantees that the field values themselves will conform to your JSON Schema.
2. When using Structured Outputs Mode, there are no guarantees that the field values themselves will conform to your JSON Schema, beyond primitive data types such as strings, numbers, booleans, etc.
3. There are no guarantees that the field values are correct (vs. being hallucinated).
4. All fields are extracted in a single inference call, with additional fields adding linearly or greater to extraction latency.

SDE’s Preprocessing, Guided LLM Output, and Validation

Zep uses a combination of dialog preprocessing, guided LLM output, and post-inference validation to ensure that the extracted data is in the format you expect and is valid given the current dialog. When using a structured Field Type (such as ZepDate, ZepEmail, ZepRegex), you will not receive back data in an incorrect format.

While there are limits to the accuracy of extraction when the conversation is very nuanced or ambiguous, with careful crafting of field descriptions, you can achieve high accuracy in most cases.

Concurrent Extraction Scales Sub-Linearly

SDE’s extraction latency scales sub-linearly with the number of fields in your model. That is, you may add additional fields with low marginal increase in latency. You can expect extraction times of 400ms or lower when extracting fairly complex models for a 500 character dialog (which includes both message content and your Role and RoleType designations).

Defining Your Model

To extract data with Zep, you will need to define a model of the data you require from a Chat History. Each model is composed of a set of fields, each of which has a type and description. Key to successful extraction of data is careful construction of the field description.

1
from pydantic import Field
2
from zep_cloud.extractor import ZepModel, ZepText, ZepEmail, ZepDate
3
4
class SalesLead(ZepModel):
5
    company_name: Optional[ZepText] = Field(
6
        description="The company name", 
7
        default=None
8
    )
9
    lead_name: Optional[ZepText] = Field(
10
        description="The lead's name", 
11
        default=None
12
    )
13
    lead_email: Optional[ZepEmail] = Field(
14
        description="The lead's email", 
15
        default=None
16
    )
17
    lead_phone: Optional[ZepPhoneNumber] = Field(
18
        description="The lead's phone number", 
19
        default=None
20
    )
21
    budget: Optional[ZepFloat] = Field(
22
        description="The lead's budget for the product", 
23
        default=None
24
    )
25
    product_name: Optional[ZepRegex] = Field(
26
        description="The name of the product the lead is interested in",
27
        pattern=r"(TimeMachine|MagicTransporter)", 
28
        default=None
29
    )
30
    zip_code: Optional[ZepZipCode] = Field(
31
        description="The company zip code", 
32
        default=None
33
    )

Executing an Extraction

To execute an extraction, you will need to call the extract method on the memory client. This method requires a session_id and a model schema that specifies the types and structures of data to be extracted based on field descriptions.

The lastN parameter, or Python equivalent last_n, specifies the number prior messages in the Session’s Chart History to look back at for data extraction.

The validate parameter specifies whether to optionally run an additional validation step on the extracted data.

The currentDateTime parameter, or Python equivalent current_date_time, specifies your user’s current date and time. This is used when extracting dates and times from relative phrases like “yesterday” or “last week” and to correctly set the timezone of the extracted data.

1
extracted_data: SalesLead = client.memory.extract(
2
    session_id,
3
    SalesLead,
4
    last_n=8,
5
    validate=False,
6
    current_date_time=datetime.now(ZoneInfo('America/New_York'))
7
)

Using Progressive Data Extraction To Guide LLMs

Your application may need to collect a number of fields in order to accomplish a task. You can guide the LLM through this process by calling extract on every chat turn, identifying which fields are still needed, providing a partially populated model to the LLM, and directing the LLM to collect the remaining data.

1
You have already collected the following data:
2
- Company name: Acme Inc.
3
- Lead name: John Doe
4
- Lead email: [email protected]
5
6
You still need to collect the following data:
7
- Lead phone number
8
- Lead budget
9
- Product name
10
- Zip code
11
12
Do not ask for all fields at once. Rather, work the fields
13
into your conversation with the user and gradually collect the data.

As each field is populated, you may copy these values into an immutable data structure. Alternatively, if existing values change as the conversation progresses, you can apply a heuristic informed by your business rules to update the data structure with the new values.

Supported Field Types

Zep supports a wide variety of field types natively. Where Zep does not support a native field type, you can use a ZepRegex field to extract a string that matches a structure you define.

Improving Accuracy

Extraction accuracy may be improved by experimenting with different descriptions and using Zep’s built-in field validation.

Improving Descriptions

When describing fields, ensure that you’ve been both specific and clear as to what value you’d like to extract. You may also provide few-shot examples in your description.

Validating Extracted Data

When validation is enabled on your extract call, Zep will run an additional LLM validation step on the extracted data. This provides improved accuracy and reduces the risk of hallucinated values. The downside to enabling field validation is increased extraction latency and an increased risk of false negatives (empty fields where the data may be present in the dialog).

We recommend running without field validation first to gauge accuracy and latency and only enable field validation if you’ve determined that it is needed given your use case.

Working with Dates

Zep understands a wide variety of date and time formats, including relative times such as “yesterday” or “last week”. It is also able to parse partial dates and times, such as “at 3pm” or “on the 15th”. All dates and times are returned in ISO 8601 format and use the timezone of the currentDateTime parameter passed to the extract call.

If you are extracting datetime and date fields it is important that you provide a currentDateTime value in your extract call and ensure that it is in the correct timezone for your user (or the base timezone your application uses internally).

Extracting from Speech Transcripts

Zep is able to understand and extract data from machine-translated transcripts. Spelled out numbers and dates will be parsed as if written language. Utterances such as “uh” or “um” are ignored.

Multilingual Data Support

Zep’s Structured Data Extraction supports most major languages.

Tips, Tricks, and Best Practices

Limit the number of Messages from which you extract data

If your use case is latency sensitive, limit the number of messages from which you extract data. The higher the last N messages, the longer the extraction will take.

Always make fields optional in Python models

Always make fields optional in your Python model. This will prevent runtime errors when the data is not present in the conversation.

Using Regex when Zep doesn’t support your data type

The ZepRegex field type is a swiss army knife for extracting data. It allows you to extract any string that matches a regex pattern defined by you.

1
class OrderInfo(ZepModel):
2
    order_id: Optional[ZepRegex] = Field(
3
        description="The order ID in format ABC-12345",
4
        pattern=r"[A-Z]{3}-\d{5}"
5
    )

Implementing Enum Fields

The ZepRegex field type can be used to extract data from a list of enums provided in a capture group.

1
order_currency: Optional[ZepRegex] = Field(
2
    description="The order currency: USD, GBP, or UNKNOWN",
3
    default=None,
4
    pattern=r"(UNKNOWN|USD|GBP)"
5
)

Results in:

"USD"

Comma Separated Lists

You can extract comma separated lists using the ZepRegex field type:

1
brand_preferences: Optional[ZepRegex] = Field(
2
    description="The customer's preferred brands as a comma-separated list",
3
    default=None,
4
    pattern=r"\w+(, \w+)+"
5
)

Results in:

"Nike, Adidas, Puma"

Unsupported Regex Patterns

The following Regex tokens and features are unsupported when using the Regex field type:

• Start of and end of string anchors (^ and $) and absolute positioning (\A and \Z).
• Named groups ((?P<name>...)).
• Backreferences (\g<name>).
• Lookaheads and lookbehinds ((?=...), (?!...), (?<=...), (?<!...)).
• Conditional expressions ((?(condition)yes|no)).