> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://help.getzep.com/v2/graphiti/configuration/llm-configuration/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://help.getzep.com/_mcp/server.

# LLM Configuration

> **Note**
>
> Graphiti works best with LLM services that support Structured Output (such as OpenAI and Gemini). Using other services may result in incorrect output schemas and ingestion failures, particularly when using smaller models.

Graphiti defaults to using OpenAI for LLM inference and embeddings, but supports multiple LLM providers including Azure OpenAI, Google Gemini, Anthropic, Groq, and local models via Ollama. This guide covers configuring Graphiti with alternative LLM providers.

## Azure OpenAI

Graphiti supports Azure OpenAI through the Azure OpenAI v1 API compatibility
layer. Use your Azure deployment names as the model names.

### Installation

```bash
pip install graphiti-core
```

### Configuration

```python
from openai import AsyncOpenAI
from graphiti_core import Graphiti
from graphiti_core.llm_client.azure_openai_client import AzureOpenAILLMClient
from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.embedder.azure_openai import AzureOpenAIEmbedderClient
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient

api_key = "<your-api-key>"
endpoint = "https://<your-resource-name>.openai.azure.com"
model_deployment = "<your-model-deployment>"
small_model_deployment = "<your-small-model-deployment>"
embedding_deployment = "<your-embedding-deployment>"

azure_client = AsyncOpenAI(
    base_url=f"{endpoint}/openai/v1/",
    api_key=api_key,
)

azure_llm_config = LLMConfig(
    model=model_deployment,
    small_model=small_model_deployment,
)

graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password",
    llm_client=AzureOpenAILLMClient(
        azure_client=azure_client,
        config=azure_llm_config,
    ),
    embedder=AzureOpenAIEmbedderClient(
        azure_client=azure_client,
        model=embedding_deployment,
    ),
    cross_encoder=OpenAIRerankerClient(
        client=azure_client,
        config=LLMConfig(model=small_model_deployment),
    ),
)
```

The maintained Azure OpenAI example reads `AZURE_OPENAI_ENDPOINT`,
`AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_DEPLOYMENT`, and
`AZURE_OPENAI_EMBEDDING_DEPLOYMENT`. Load these variables in your application,
and pass their values to the clients as shown above.

This example uses one Azure OpenAI resource. If the chat and embedding
deployments use different resources, create a separate `AsyncOpenAI` client for
each resource. Pass the chat client to `AzureOpenAILLMClient` and
`OpenAIRerankerClient`. Pass the embedding client to
`AzureOpenAIEmbedderClient`.

## Google Gemini

Google's Gemini models support structured output and can be used for LLM inference, embeddings, and cross-encoding/reranking.

### Installation

```bash
pip install "graphiti-core[google-genai]"
```

### Configuration

```python
from graphiti_core import Graphiti
from graphiti_core.llm_client.gemini_client import GeminiClient, LLMConfig
from graphiti_core.embedder.gemini import GeminiEmbedder, GeminiEmbedderConfig
from graphiti_core.cross_encoder.gemini_reranker_client import GeminiRerankerClient

# Google API key configuration
api_key = "<your-google-api-key>"

# Initialize Graphiti with Gemini clients
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password",
    llm_client=GeminiClient(
        config=LLMConfig(
            api_key=api_key,
            model="gemini-3.7-flash"
        )
    ),
    embedder=GeminiEmbedder(
        config=GeminiEmbedderConfig(
            api_key=api_key,
            embedding_model="embedding-001"
        )
    ),
    cross_encoder=GeminiRerankerClient(
        config=LLMConfig(
            api_key=api_key,
            model="gemini-3.7-flash"
        )
    )
)
```

This example uses `gemini-3.7-flash` for generation and reranking.

### Environment Variables

Google Gemini can be configured using:

* `GOOGLE_API_KEY` - Your Google API key

## Anthropic

Anthropic's Claude models can be used for LLM inference with OpenAI embeddings and reranking.

> **Warning**
>
> When using Anthropic for LLM inference, you still need an OpenAI API key for embeddings and reranking functionality. Make sure to set both `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` environment variables.

### Installation

```bash
pip install "graphiti-core[anthropic]"
```

### Configuration

```python
from graphiti_core import Graphiti
from graphiti_core.llm_client.anthropic_client import AnthropicClient, LLMConfig
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient

# Configure Anthropic LLM with OpenAI embeddings and reranking
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j", 
    "password",
    llm_client=AnthropicClient(
        config=LLMConfig(
            api_key="<your-anthropic-api-key>",
            model="claude-sonnet-4-5-latest",
            small_model="claude-haiku-4-5-latest"
        )
    ),
    embedder=OpenAIEmbedder(
        config=OpenAIEmbedderConfig(
            api_key="<your-openai-api-key>",
            embedding_model="text-embedding-3-small"
        )
    ),
    cross_encoder=OpenAIRerankerClient(
        config=LLMConfig(
            api_key="<your-openai-api-key>",
            model="gpt-4.1-nano"
        )
    )
)
```

### Environment Variables

Anthropic can be configured using:

* `ANTHROPIC_API_KEY` - Your Anthropic API key
* `OPENAI_API_KEY` - Required for embeddings and reranking

## Groq

Groq provides fast inference with various open-source models, using OpenAI for embeddings and reranking.

> **Warning**
>
> When using Groq, avoid smaller models as they may not accurately extract data or output the correct JSON structures required by Graphiti. Use larger, more capable models like Llama 3.1 70B for best results.

### Installation

```bash
pip install "graphiti-core[groq]"
```

### Configuration

```python
from graphiti_core import Graphiti
from graphiti_core.llm_client.groq_client import GroqClient, LLMConfig
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient

# Configure Groq LLM with OpenAI embeddings and reranking
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password", 
    llm_client=GroqClient(
        config=LLMConfig(
            api_key="<your-groq-api-key>",
            model="llama-3.1-70b-versatile",
            small_model="llama-3.1-8b-instant"
        )
    ),
    embedder=OpenAIEmbedder(
        config=OpenAIEmbedderConfig(
            api_key="<your-openai-api-key>",
            embedding_model="text-embedding-3-small"
        )
    ),
    cross_encoder=OpenAIRerankerClient(
        config=LLMConfig(
            api_key="<your-openai-api-key>",
            model="gpt-4.1-nano"
        )
    )
)
```

### Environment Variables

Groq can be configured using:

* `GROQ_API_KEY` - Your Groq API key
* `OPENAI_API_KEY` - Required for embeddings

## Ollama (Local LLMs)

Ollama enables running local LLMs and embedding models via its OpenAI-compatible API, ideal for privacy-focused applications or avoiding API costs.

> **Warning**
>
> When using Ollama, avoid smaller local models as they may not accurately extract data or output the correct JSON structures required by Graphiti. Use larger, more capable models and ensure they support structured output for reliable knowledge graph construction.

> **Note**
>
> Ollama provides an OpenAI-compatible API, but does not support the `/v1/responses` endpoint that `OpenAIClient` uses. Use `OpenAIGenericClient` instead, which uses the `/v1/chat/completions` endpoint with `response_format` for structured outputs—both of which Ollama supports.

### Installation

First, install and configure Ollama:

```bash
# Install Ollama (visit https://ollama.ai for installation instructions)
# Then pull the models you want to use:
ollama pull deepseek-r1:7b     # LLM
ollama pull nomic-embed-text   # embeddings
```

### Configuration

```python
from graphiti_core import Graphiti
from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.llm_client.openai_generic_client import OpenAIGenericClient
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient

# Configure Ollama LLM client using OpenAIGenericClient for compatibility
llm_config = LLMConfig(
    api_key="ollama",  # Ollama doesn't require a real API key
    model="deepseek-r1:7b",
    small_model="deepseek-r1:7b",
    base_url="http://localhost:11434/v1",
)

llm_client = OpenAIGenericClient(config=llm_config)

# Initialize Graphiti with Ollama clients
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password",
    llm_client=llm_client,
    embedder=OpenAIEmbedder(
        config=OpenAIEmbedderConfig(
            api_key="ollama",
            embedding_model="nomic-embed-text",
            embedding_dim=768,
            base_url="http://localhost:11434/v1",
        )
    ),
    cross_encoder=OpenAIRerankerClient(client=llm_client.client, config=llm_config),
)
```

Ensure Ollama is running (`ollama serve`) and that you have pulled the models you want to use.

## OpenAI Compatible Services

Many LLM providers offer OpenAI-compatible APIs. Use the `OpenAIGenericClient` for these services, which ensures proper schema injection for JSON output since most providers don't support OpenAI's structured output format.

> **Warning**
>
> When using OpenAI-compatible services, avoid smaller models as they may not accurately extract data or output the correct JSON structures required by Graphiti. Choose larger, more capable models that can handle complex reasoning and structured output.

### Installation

```bash
pip install graphiti-core
```

### Configuration

```python
from graphiti_core import Graphiti
from graphiti_core.llm_client.openai_generic_client import OpenAIGenericClient
from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
from graphiti_core.cross_encoder.openai_reranker_client import OpenAIRerankerClient

# Configure OpenAI-compatible service
llm_config = LLMConfig(
    api_key="<your-api-key>",
    model="<your-main-model>",        # e.g., "mistral-large-latest"
    small_model="<your-small-model>", # e.g., "mistral-small-latest"
    base_url="<your-base-url>",       # e.g., "https://api.mistral.ai/v1"
)

# Initialize Graphiti with OpenAI-compatible service
graphiti = Graphiti(
    "bolt://localhost:7687",
    "neo4j",
    "password",
    llm_client=OpenAIGenericClient(config=llm_config),
    embedder=OpenAIEmbedder(
        config=OpenAIEmbedderConfig(
            api_key="<your-api-key>",
            embedding_model="<your-embedding-model>", # e.g., "mistral-embed"
            base_url="<your-base-url>",
        )
    ),
    cross_encoder=OpenAIRerankerClient(
        config=LLMConfig(
            api_key="<your-api-key>",
            model="<your-small-model>",  # Use smaller model for reranking
            base_url="<your-base-url>",
        )
    )
)
```

Replace the placeholder values with your actual service credentials and model names.

> Zep unifies business data, documents, and conversations into shared, governed context that agents can retrieve for their tasks.