Prepare Data for Ingestion

Good ingestion starts before the API call. Normalize entity names, preserve source context and timestamps, and decide which entities or facts must be represented exactly. These choices reduce duplicate nodes and improve extraction and retrieval.

Three terms recur below. An episode is one unit of source material you send to Zep, such as a document chunk, a message, or a JSON record, from which Zep extracts entities and facts. A node is one entity in the resulting graph, such as a person, team, or account. An ontology is the set of entity and relationship types Zep looks for while extracting.

These practices apply whether you call the Zep SDK directly or use zep-ingest, a Python package of loaders, transforms, and monitoring helpers built on the Zep SDK. When zep-ingest provides a built-in feature for a recommendation, this guide identifies the specific helper to use. zep-ingest 0.1 is alpha software, so pin its version, and use the SDK directly if your import cannot absorb interface changes.

Know the limits before you design the import

Exceeding any of these is a rejection, not degraded quality. Two of them constrain how you model your data rather than how you make a single call, so check them before you write ingestion code.

Exceeding a payload limit returns a 400 Bad Request error. Chunk documents that are larger than the episode limit before submitting them; see Chunking large documents.

The entity-type and edge-type caps are counted separately rather than as a combined total, so 10 entity types and 10 edge types together are accepted. Enterprise and Flex Plus accounts are not subject to the 10-type caps, but no account can set more than 20 entity types or 20 edge types.

Use one canonical name per entity

When Zep extracts entities from episodes, it deduplicates them primarily by name. It first checks exact names after normalizing case and whitespace, then may use fuzzy similarity and model-based resolution. These later checks are heuristic and do not guarantee a merge.

Rewrite known aliases to one canonical name before ingestion. If Sarah and Sarah Brown refer to the same person, use Sarah Brown consistently. Otherwise, Zep may create separate nodes for them.

Choose a canonical name from an authoritative source such as your user directory or CRM. An alias map takes no scope argument, so the unit of scoping is the pipeline: build one map per source and process each source separately, as shown below.

Canonicalize aliases with zep-ingest

If you use zep-ingest, the package includes AliasCanonicalizer for replacing known aliases before Zep extracts entities. Add it as a pipeline transform when names have not already been normalized upstream.

1
from zep_ingest import AliasCanonicalizer
2
3
canonicalize_people = AliasCanonicalizer(
4
    {
5
        "Sarah Brown": ["Sarah B.", "S. Brown"],
6
        "Robert Chen": ["Bob Chen"],
7
    }
8
)

The default rewrite mode replaces aliases with their canonical value. Use mode="annotate" when preserving the original text is important:

1
canonicalize_people = AliasCanonicalizer(
2
    {"Sarah Brown": ["Sarah B."]},
3
    mode="annotate",
4
)

AliasCanonicalizer behavior and safeguards

• Matching is case-sensitive and uses word boundaries by default.
• URLs, inline code, and existing canonical names are not rewritten.
• An alias cannot map to two different canonical names in the same transform.
• JSON episodes are passed through unchanged. Normalize aliases in structured fields before loading them, or use a custom transform.
• The default safety guard rejects aliases shorter than three characters and many common words or ambiguous given names.

For example, do not map the bare alias Will to William Hughes. Will can be a person’s name, but it is also a common word in sentences such as “Will review the proposal.” Sentence-start capitalization makes that unsafe even with case-sensitive matching. Use an unambiguous alias such as Will Hughes:

1
canonicalize_people = AliasCanonicalizer(
2
    {
3
        "William Hughes": ["Will Hughes", "W. Hughes"],
4
    }
5
)

The default risky-word list includes names and words such as Will, May, Mark, Bill, Art, and Page. You can extend it with terms that are ambiguous in your domain:

1
from zep_ingest import AliasCanonicalizer, DEFAULT_RISKY_WORDS
2
3
canonicalize_people = AliasCanonicalizer(
4
    {"William Hughes": ["Will Hughes"]},
5
    risky_words=DEFAULT_RISKY_WORDS | {"lead", "support"},
6
)

Give each source its own alias map, then preview the complete import and review the per-alias replacement counts before submitting anything. preview() makes no Zep API calls, and limit=None validates the whole stream rather than a sample:

1
from zep_ingest import AliasCanonicalizer, Pipeline, TextChunker, TextFileLoader
2
3
PER_SOURCE_ALIASES = {
4
    "exports/crm/**/*.md": {"Sarah Brown": ["Sarah B.", "S. Brown"]},
5
    "exports/support/**/*.md": {"Sarah Okafor": ["Sarah O."]},
6
}
7
8
for pattern, aliases in PER_SOURCE_ALIASES.items():
9
    pipeline = Pipeline(
10
        TextFileLoader(pattern),
11
        transforms=[
12
            AliasCanonicalizer(aliases),
13
            TextChunker(chunk_size=500, overlap=50),
14
        ],
15
    )
16
17
    for warning in pipeline.preview(limit=None).warnings:
18
        print(warning)

An exhaustive preview reads every source file and holds the transformed episodes in memory, so allow time and memory proportional to the size of the export.

Set up an import manifest before your first write

Reingesting the same file creates new episodes, even when its contents are unchanged. zep-ingest does not currently deduplicate episodes by source record or content, and it cannot resume a run from the middle of a source. The identifiers that make an import recoverable are only available at the moment you submit, so the manifest has to exist before the first write rather than after the first failure.

Maintain an import manifest outside Zep until zep-ingest provides resume and idempotency support. At minimum, record:

• destination graph or user ID;
• source-system record ID;
• normalized content hash;
• source modification time or export version;
• import run ID;
• returned Batch IDs, episode UUIDs, and task IDs;
• terminal status.

Before submitting a record, compare its source ID and content hash with the last successful import. If the record changed, decide whether to add new historical evidence, update a directly managed node, or delete and replace earlier data. Deleting data from the graph describes what removing an episode, node, or thread takes with it.

If an import stops partway through, the manifest is what tells you which records to resubmit; see Recover from a partial import.

Use stable identifiers when available

Canonical names improve extraction, but stable identifiers are the safer choice for data that must not duplicate.

Add direct nodes with zep-ingest

A direct node UUID is a stable identifier that your application assigns to one graph entity. Zep uses it like a database primary key. Reusing the UUID updates the same node; using another UUID creates another node even when the name is identical. During an upsert, Zep preserves the existing name, replaces supplied summary or type values, and merges supplied attributes. Use graph.node.update when you need to rename a node, delete an attribute, or clear its entity type. Manually updating the graph covers the direct node and fact-triple operations.

If you use zep-ingest, create each direct node with the built-in NodeItem model and submit it with ingest_nodes. The helper requires a persisted UUIDv4 by default so the import can safely upsert the same node on a later run.

NodeItem accepts version 4 UUIDs only. A deterministic UUIDv5 derived from your source ID is rejected when the item is constructed, so deriving IDs is not a way to avoid keeping the mapping. Generate a UUIDv4 once per source record and store it.

Suppose your employee directory identifies Sarah as EMP-1842. Generate a UUID once, store the mapping from EMP-1842 to that UUID, and reuse it on every import:

1
import os
2
3
from zep_cloud.client import Zep
4
from zep_ingest import NodeItem, ingest_nodes
5
6
client = Zep(api_key=os.environ["ZEP_API_KEY"])
7
sarah_uuid = "c2b42bf8-5d3f-4c92-92a6-43d8854a1e5b"
8
9
# First import: create Sarah's node.
10
created = ingest_nodes(
11
    client,
12
    [
13
        NodeItem(
14
            uuid=sarah_uuid,
15
            name="Sarah Brown",
16
            label="Person",
17
            attributes={
18
                "employee_id": "EMP-1842",
19
                "title": "Product Manager",
20
            },
21
        )
22
    ],
23
    graph_id="company-kb",
24
)
25
created.wait(timeout=600)
26
27
# Later import: the same UUID updates the same node.
28
updated = ingest_nodes(
29
    client,
30
    [
31
        NodeItem(
32
            uuid=sarah_uuid,
33
            name="Sarah Brown",
34
            label="Person",
35
            attributes={
36
                "employee_id": "EMP-1842",
37
                "title": "VP of Product",
38
            },
39
        )
40
    ],
41
    graph_id="company-kb",
42
)
43
updated.wait(timeout=600)

If the second import uses a newly generated UUID, Zep creates another Sarah Brown node. ingest_nodes therefore requires persisted UUIDs by default. graph.add_nodes assigns a UUID when one is omitted, but that assigned UUID must be saved and reused if later imports should update the same node.

For fact triples, pass source_node_uuid and target_node_uuid when the endpoints already exist. If you omit an endpoint UUID, Zep resolves the node by name and may create a node when no match is found.

Configure identity properties with the Zep API

Identity properties let Zep match extracted entities using an exact, type-specific value before considering name similarity. They are useful when two sources use different display names but share a value such as a company domain.

Identity properties are part of the Zep ontology API, not a zep-ingest transform. Configure them on the destination ontology before using zep-ingest to submit episodes, messages, or documents.

For example, two episodes may describe the same company differently:

1
{"name": "Acme", "domain": "acme.com"}
2
{"name": "Acme Incorporated", "domain": "acme.com"}

Define domain as the identity property for the Company entity type:

1
from zep_cloud.types import EntityProperty, EntityType
2
3
company = EntityType(
4
    name="Company",
5
    description="A company or legal organization",
6
    properties=[
7
        EntityProperty(
8
            name="domain",
9
            type="Text",
10
            description="The company's primary domain",
11
        )
12
    ],
13
    identity_properties=["domain"],
14
)

A property’s type is one of Text, Int, Float, or Boolean.

If Zep classifies both extracted entities as Company and extracts domain on both, the exact acme.com identity match resolves them to the same node before name similarity or model-based resolution runs. If a record omits the domain or supplies a different value such as www.acme.com, the identity property does not match. Normalize identity values upstream.

Every identity property must be declared on the entity type. Multiple properties use AND semantics: all values must be present and match. Missing, null, empty, object, and array values do not form a match. String comparisons ignore surrounding whitespace but are otherwise exact.

Customizing graph structure covers entity and edge type modelling through EntityModel subclasses, which do not carry identity properties.

Include context in every episode

Put this context in the episode text itself. Zep extracts entities and facts from the text. Metadata is projected onto the artifacts derived from an episode and lets you filter graph search results, but it is not a place to keep context you want extracted.

An episode should be understandable without relying on the previous record or a filename that Zep never receives.

Preserve context that identifies:

• who said or authored the content;
• the channel, thread, meeting, document, or email subject;
• the organization, account, project, or product involved;
• the source record ID and system;
• the original event time.

For a transcript, prefer:

Meeting: Weekly launch review
Date: 2025-04-18
Sarah Brown (Product): The launch date moved to May 10.
Robert Chen (Support): I will update the customer notice by Friday.

over:

The team discussed the launch. It moved, and someone will update customers.

The second version removes speaker attribution, the exact date, and the owner of the commitment.

The zep-ingest package’s SlackExportLoader, TranscriptLoader, and EmlLoader preserve source-specific context inline. For a custom loader, add the equivalent context to the text of every episode.

Shape each source for extraction

Chunk documents and transcripts with zep-ingest

Zep rejects an episode larger than 10,000 characters rather than truncating it, so any longer document has to be split. Split at meaningful boundaries. Keep headings or source context with each chunk. For transcripts, preserve complete speaker turns where possible.

If you use zep-ingest, the built-in TextChunker uses paragraph and sentence boundaries with configurable overlap:

1
from zep_ingest import TextChunker
2
3
chunker = TextChunker(chunk_size=500, overlap=50)

chunk_size and overlap are counted in characters, not tokens or words. Do not chunk records that are already one self-contained unit at a natural size, such as a short support ticket or a CRM note; splitting them only breaks context that was already correct.

The package’s optional LLMContextualizer can prepend source context to each chunk. It sends the chunk and document context to the LLM client you configure. Review that provider’s data-handling terms before enabling it.

Shape JSON records before loading them

JSON works best when each episode is shallow, self-contained, and centered on one entity. Whether to split a wide object, expand a long list, flatten deep nesting, or pull a long string out as text depends on what the data means, so these are decisions you make about your own records rather than work a generic transform can do for you. Repeat fields such as id, name, and description when one source record becomes several episodes, so every piece still names the entity it describes. Adding JSON best practices sets out the full shaping rules.

If you use zep-ingest, the built-in ingest_json_records helper submits each record exactly as you provide it, one JSON episode per record, and applies no shaping transforms. What it adds is identity and context: it reads CSV, JSONL, and JSON arrays, maps your own fields onto id, name, and description, lifts a timestamp field into created_at, tags each record with a record type, and copies the fields you name in metadata_fields into episode metadata. It does reject rather than reshape: because NaN, Infinity, and -Infinity are not valid JSON, a record containing any of them is refused with the file, record number, and field path named.

1
from zep_ingest import ingest_json_records
2
3
result = ingest_json_records(
4
    client,
5
    "customers.jsonl",
6
    graph_id="customer-accounts",
7
    id_field="customer_id",
8
    name_field="company_name",
9
    description_field="account_summary",
10
    created_at_field="updated_at",
11
    metadata_fields=["source_system"],
12
    record_type="customer_account",
13
)

So that a fact can be traced back to where it came from, zep-ingest also stamps provenance on every episode it produces: a source_type naming the kind of source, plus the identifying detail for that kind. For ingest_json_records that is a source_type of json_record and the file_name the record was read from. Because those two keys are reserved, metadata_fields can name at most eight fields per record. A ninth raises ConfigurationError when the loader is constructed, before any file is read:

metadata_fields names 9 fields; at most 8 are allowed. Every episode reserves 2 of the API's 10 metadata keys for provenance (source_type, file_name).

Naming one of the two reserved keys in metadata_fields is a warning rather than an error, so provenance stays trustworthy: a record’s own source_type or file_name is not lifted, and that value remains only in the episode body.

LimitGuard still keeps every episode under the size limit, splitting an oversized record at its top level and warning you. That is a safeguard against rejection, not a shaping step: the pieces it produces can lose the cross-references that held the record together, so split large records yourself rather than relying on it.

The pieces are numbered by a part metadata marker. LimitGuard omits that marker, and warns, in two cases: when the episode already carries ten metadata keys, and when it already has a part key of its own. In the second case your own value is kept rather than overwritten.

Preserve original event time

Supply created_at for episodes and messages using the original event time in RFC3339 format. If it is absent, Zep uses ingestion time.

This distinction matters for backfills. Data imported today may describe events from several years ago. Dating all of it today changes the order Zep uses when determining which facts were valid later.

Use the most authoritative timestamp for the source:

Do not use filesystem modification time unless it represents the source event. The zep-ingest package’s TextFileLoader uses it only when you explicitly set use_file_mtime=True.

Choose extraction or direct assertions

Use episodes when the source is material Zep should interpret. Use explicit nodes and fact triples when your application already knows the answer.

If you use zep-ingest, use ingest_nodes for known entities and ingest_fact_triples for known relationships. The package’s document, conversation, email, transcript, Slack, and JSON helpers create episodes or messages for Zep to interpret.

Examples that often require an exact assertion include:

• approved decisions;
• assigned action items;
• contractual commitments;
• the current owner of a system;
• authoritative account or employee relationships;
• a correction that supersedes tentative discussion.

A conversation can contain proposals, disagreements, and abandoned plans. Extraction cannot infer your application’s approval or authority policy unless that policy is represented in the data. Normalize those states before ingestion, or write the approved relationship as a fact triple with the correct valid_at.

When combining both paths, use this order:

1. Create the destination.
2. Set its ontology. An ontology applies only to data processed after it is set, and existing data is not re-extracted or retyped automatically, so a type you add later does not reach anything already ingested. Setting an ontology also replaces the whole type set for that scope, so send the complete list every time rather than only the types you are adding.
3. Seed known nodes or facts.
4. Ingest historical documents and conversations in chronological order.
5. Apply current known assertions with their real validity times.

Processing complete does not mean search ready

There are three separate checkpoints:

1. Submission accepted: the API accepted the episodes, messages, nodes, or facts.
2. Processing complete: extraction or graph mutation reached a terminal state.
3. Search ready: the resulting graph data is available to search.

If you use zep-ingest, every ingestion helper returns an IngestResult. Use IngestResult.wait() for the second checkpoint when the result contains Batch, episode, or task completion handles. When Zep accepts a write but returns no handle for it, the result counts those items as untracked and wait() raises IngestUntrackedError, so confirming them needs a read of your own.

The package’s search_when_ready() helper retries until a query returns any result or reaches the timeout. It does not verify that a specific imported record produced the result. Use a query unique to the imported data when checking indexing readiness. If processing succeeded but expected results remain absent, inspect the graph, source context, timestamps, and query before reingesting.