Bring Your Own LLM (BYOM)

Overview

Bring Your Own LLM (BYOM) lets you connect your existing contracts with model providers such as OpenAI, Anthropic, and Google to Zep Cloud. You keep using Zep’s orchestration, memory, and security controls while routing inference through credentials you manage. This approach ensures:

• Contract continuity: Apply your negotiated pricing, quotas, and compliance commitments with each LLM vendor.
• Data governance: Enforce provider-specific policies for data usage, retention, and residency.
• Operational flexibility: Configure the best vendor or model for each project, including fallbacks for high availability.

Getting started

1. Collect provider credentials. Obtain API keys or service accounts for your chosen vendors. Each Zep project can use a different set of credentials, enabling separation between environments.
2. Add credentials in the Zep dashboard. Navigate to Settings ▸ LLM Providers within a project, select a vendor, and paste the credential. Zep stores the secret securely in an encrypted secrets manager within your project scope.
3. (Optional) Supply a customer-managed KMS key. If you require customer-controlled encryption, provide a KMS ARN with kms:Encrypt, kms:Decrypt, and kms:DescribeKey permissions granted to Zep’s runtime roles. Zep validates the key with a test encrypt/decrypt during setup.
4. Select default and fallback models. Choose a primary model for the project. Optionally configure fallbacks to maintain continuity if the primary vendor rate limits or experiences an outage.
5. Monitor usage and quotas. Use project analytics to track call volume by provider. Configure per-provider rate limits to enforce budget or vendor restrictions.

FAQ

Does Zep store our provider keys in its databases?
No. Keys are stored securely in an encrypted secrets manager. Values are decrypted in memory only when needed and are never written to Zep databases.

Can we use different vendors or models per project?
Yes. Each project maintains its own provider configuration, including defaults and fallbacks. This is useful for isolating production from staging or testing providers side by side.

Can we prevent vendors from training on our data?
Yes. Use the vendor endpoints and contractual controls that disable data retention or training. Zep routes requests accordingly and sets the necessary flags in each call.

How is usage billed?
You receive invoices from Zep for Zep services only. LLM inference charges come directly from your vendors under your existing contract and pricing.

What happens if a key is compromised or needs rotation?
Add a new credential in the dashboard, mark it as active, then disable the previous one. Requests start using the new credential immediately; no downtime is required.

How does BYOM affect observability?
Requests are tagged by project and provider, so you can attribute usage and costs. Rate limits can be applied per provider to protect budgets and enforce quotas.