LLM Configuration | Zep Documentation

Graphiti works best with LLM services that support Structured Output (such as OpenAI and Gemini). Using other services may result in incorrect output schemas and ingestion failures, particularly when using smaller models.

Graphiti defaults to using OpenAI for LLM inference and embeddings, but supports multiple LLM providers including Azure OpenAI, Google Gemini, Anthropic, Groq, and local models via Ollama. This guide covers configuring Graphiti with alternative LLM providers.

Azure OpenAI

Graphiti supports Azure OpenAI through the Azure OpenAI v1 API compatibility layer. Use your Azure deployment names as the model names.

Installation

pip install graphiti-core

Configuration

from openai import AsyncOpenAI
from graphiti_core import Graphiti
from graphiti_core.llm_client.azure_openai_client import AzureOpenAILLMClient
from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.embedder.azure_openai import AzureOpenAIEmbedderClient
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient
api_key = "<your-api-key>"
endpoint = "https://<your-resource-name>.openai.azure.com"
model_deployment = "<your-model-deployment>"
small_model_deployment = "<your-small-model-deployment>"
embedding_deployment = "<your-embedding-deployment>"
azure_client = AsyncOpenAI(
    base_url=f"{endpoint}/openai/v1/",
    api_key=api_key,
)
azure_llm_config = LLMConfig(
    model=model_deployment,
    small_model=small_model_deployment,
)
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password",
    llm_client=AzureOpenAILLMClient(
        azure_client=azure_client,
        config=azure_llm_config,
    ),
    embedder=AzureOpenAIEmbedderClient(
        azure_client=azure_client,
        model=embedding_deployment,
    ),
    cross_encoder=OpenAIRerankerClient(
        client=azure_client,
        config=LLMConfig(model=small_model_deployment),
    ),
)

The maintained Azure OpenAI example reads AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, AZURE_OPENAI_DEPLOYMENT, and AZURE_OPENAI_EMBEDDING_DEPLOYMENT. Load these variables in your application, and pass their values to the clients as shown above.

This example uses one Azure OpenAI resource. If the chat and embedding deployments use different resources, create a separate AsyncOpenAI client for each resource. Pass the chat client to AzureOpenAILLMClient and OpenAIRerankerClient. Pass the embedding client to AzureOpenAIEmbedderClient.

Google Gemini

Google’s Gemini models support structured output and can be used for LLM inference, embeddings, and cross-encoding/reranking.

Installation

pip install "graphiti-core[google-genai]"

Configuration

from graphiti_core import Graphiti
from graphiti_core.llm_client.gemini_client import GeminiClient, LLMConfig
from graphiti_core.embedder.gemini import GeminiEmbedder, GeminiEmbedderConfig
from graphiti_core.cross_encoder.gemini_reranker_client import GeminiRerankerClient
# Google API key configuration
api_key = "<your-google-api-key>"
# Initialize Graphiti with Gemini clients
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password",
    llm_client=GeminiClient(
        config=LLMConfig(
            api_key=api_key,
            model="gemini-3.7-flash"
        )
    ),
    embedder=GeminiEmbedder(
        config=GeminiEmbedderConfig(
            api_key=api_key,
            embedding_model="embedding-001"
        )
    ),
    cross_encoder=GeminiRerankerClient(
        config=LLMConfig(
            api_key=api_key,
            model="gemini-3.7-flash"
        )
    )
)

This example uses gemini-3.7-flash for generation and reranking.

Environment Variables

Google Gemini can be configured using:

GOOGLE_API_KEY - Your Google API key

Anthropic

Anthropic’s Claude models can be used for LLM inference with OpenAI embeddings and reranking.

When using Anthropic for LLM inference, you still need an OpenAI API key for embeddings and reranking functionality. Make sure to set both ANTHROPIC_API_KEY and OPENAI_API_KEY environment variables.

Installation

pip install "graphiti-core[anthropic]"

Configuration

from graphiti_core import Graphiti
from graphiti_core.llm_client.anthropic_client import AnthropicClient, LLMConfig
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient
# Configure Anthropic LLM with OpenAI embeddings and reranking
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j", 
    "password",
    llm_client=AnthropicClient(
        config=LLMConfig(
            api_key="<your-anthropic-api-key>",
            model="claude-sonnet-4-5-latest",
            small_model="claude-haiku-4-5-latest"
        )
    ),
    embedder=OpenAIEmbedder(
        config=OpenAIEmbedderConfig(
            api_key="<your-openai-api-key>",
            embedding_model="text-embedding-3-small"
        )
    ),
    cross_encoder=OpenAIRerankerClient(
        config=LLMConfig(
            api_key="<your-openai-api-key>",
            model="gpt-4.1-nano"
        )
    )
)

Environment Variables

Anthropic can be configured using:

ANTHROPIC_API_KEY - Your Anthropic API key
OPENAI_API_KEY - Required for embeddings and reranking

Groq

Groq provides fast inference with various open-source models, using OpenAI for embeddings and reranking.

When using Groq, avoid smaller models as they may not accurately extract data or output the correct JSON structures required by Graphiti. Use larger, more capable models like Llama 3.1 70B for best results.

Installation

pip install "graphiti-core[groq]"

Configuration

from graphiti_core import Graphiti
from graphiti_core.llm_client.groq_client import GroqClient, LLMConfig
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient
# Configure Groq LLM with OpenAI embeddings and reranking
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password", 
    llm_client=GroqClient(
        config=LLMConfig(
            api_key="<your-groq-api-key>",
            model="llama-3.1-70b-versatile",
            small_model="llama-3.1-8b-instant"
        )
    ),
    embedder=OpenAIEmbedder(
        config=OpenAIEmbedderConfig(
            api_key="<your-openai-api-key>",
            embedding_model="text-embedding-3-small"
        )
    ),
    cross_encoder=OpenAIRerankerClient(
        config=LLMConfig(
            api_key="<your-openai-api-key>",
            model="gpt-4.1-nano"
        )
    )
)

Environment Variables

Groq can be configured using:

GROQ_API_KEY - Your Groq API key
OPENAI_API_KEY - Required for embeddings

Ollama (Local LLMs)

Ollama enables running local LLMs and embedding models via its OpenAI-compatible API, ideal for privacy-focused applications or avoiding API costs.

When using Ollama, avoid smaller local models as they may not accurately extract data or output the correct JSON structures required by Graphiti. Use larger, more capable models and ensure they support structured output for reliable knowledge graph construction.

Ollama provides an OpenAI-compatible API, but does not support the /v1/responses endpoint that OpenAIClient uses. Use OpenAIGenericClient instead, which uses the /v1/chat/completions endpoint with response_format for structured outputs—both of which Ollama supports.

Installation

First, install and configure Ollama:

# Install Ollama (visit https://ollama.ai for installation instructions)
# Then pull the models you want to use:
ollama pull deepseek-r1:7b     # LLM
ollama pull nomic-embed-text   # embeddings

Configuration

from graphiti_core import Graphiti
from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.llm_client.openai_generic_client import OpenAIGenericClient
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient
# Configure Ollama LLM client using OpenAIGenericClient for compatibility
llm_config = LLMConfig(
    api_key="ollama",  # Ollama doesn't require a real API key
    model="deepseek-r1:7b",
    small_model="deepseek-r1:7b",
    base_url="http://localhost:11434/v1",
)
llm_client = OpenAIGenericClient(config=llm_config)
# Initialize Graphiti with Ollama clients
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password",
    llm_client=llm_client,
    embedder=OpenAIEmbedder(
        config=OpenAIEmbedderConfig(
            api_key="ollama",
            embedding_model="nomic-embed-text",
            embedding_dim=768,
            base_url="http://localhost:11434/v1",
        )
    ),
    cross_encoder=OpenAIRerankerClient(client=llm_client.client, config=llm_config),
)

Ensure Ollama is running (ollama serve) and that you have pulled the models you want to use.

OpenAI Compatible Services

Many LLM providers offer OpenAI-compatible APIs. Use the OpenAIGenericClient for these services, which ensures proper schema injection for JSON output since most providers don’t support OpenAI’s structured output format.

When using OpenAI-compatible services, avoid smaller models as they may not accurately extract data or output the correct JSON structures required by Graphiti. Choose larger, more capable models that can handle complex reasoning and structured output.

Installation

pip install graphiti-core

Configuration

from graphiti_core import Graphiti
from graphiti_core.llm_client.openai_generic_client import OpenAIGenericClient
from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient
# Configure OpenAI-compatible service
llm_config = LLMConfig(
    api_key="<your-api-key>",
    model="<your-main-model>",        # e.g., "mistral-large-latest"
    small_model="<your-small-model>", # e.g., "mistral-small-latest"
    base_url="<your-base-url>",       # e.g., "https://api.mistral.ai/v1"
)
# Initialize Graphiti with OpenAI-compatible service
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password",
    llm_client=OpenAIGenericClient(config=llm_config),
    embedder=OpenAIEmbedder(
        config=OpenAIEmbedderConfig(
            api_key="<your-api-key>",
            embedding_model="<your-embedding-model>", # e.g., "mistral-embed"
            base_url="<your-base-url>",
        )
    ),
    cross_encoder=OpenAIRerankerClient(
        config=LLMConfig(
            api_key="<your-api-key>",
            model="<your-small-model>",  # Use smaller model for reranking
            base_url="<your-base-url>",
        )
    )
)

Replace the placeholder values with your actual service credentials and model names.