181 lines
4.6 KiB
Plaintext
181 lines
4.6 KiB
Plaintext
---
|
|
title: "LiteLLM SDK"
|
|
description: "Use Bifrost as a drop-in proxy for LiteLLM applications with zero code changes."
|
|
icon: "train"
|
|
---
|
|
|
|
Since LiteLLM already provides multi-provider abstraction, Bifrost adds enterprise features like governance, semantic caching, MCP tools, observability, etc, on top of your existing setup.
|
|
|
|
**Endpoint:** `/litellm`
|
|
|
|
<Warning>
|
|
**Provider Compatibility:** This integration only works for AI providers that both LiteLLM and Bifrost support. If you're using a provider specific to LiteLLM that Bifrost doesn't support (or vice versa), those requests will fail.
|
|
</Warning>
|
|
---
|
|
|
|
## Setup
|
|
|
|
<Tabs group="litellm-sdk">
|
|
<Tab title="Python">
|
|
|
|
```python {7}
|
|
from litellm import completion
|
|
|
|
# Configure client to use Bifrost
|
|
response = completion(
|
|
model="gpt-4o-mini",
|
|
messages=[{"role": "user", "content": "Hello!"}],
|
|
base_url="http://localhost:8080/litellm" # Point to Bifrost
|
|
)
|
|
|
|
print(response.choices[0].message.content)
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
---
|
|
|
|
## Provider/Model Usage Examples
|
|
|
|
Your existing LiteLLM provider switching works unchanged through Bifrost:
|
|
|
|
<Tabs group="litellm-sdk">
|
|
<Tab title="Python">
|
|
|
|
```python {4}
|
|
from litellm import completion
|
|
|
|
# All your existing LiteLLM patterns work the same
|
|
base_url = "http://localhost:8080/litellm"
|
|
|
|
# OpenAI models
|
|
openai_response = completion(
|
|
model="gpt-4o-mini",
|
|
messages=[{"role": "user", "content": "Hello GPT!"}],
|
|
base_url=base_url
|
|
)
|
|
|
|
# Anthropic models
|
|
anthropic_response = completion(
|
|
model="claude-3-sonnet-20240229",
|
|
messages=[{"role": "user", "content": "Hello Claude!"}],
|
|
base_url=base_url
|
|
)
|
|
|
|
# Google models
|
|
google_response = completion(
|
|
model="gemini/gemini-1.5-flash",
|
|
messages=[{"role": "user", "content": "Hello Gemini!"}],
|
|
base_url=base_url
|
|
)
|
|
|
|
# Azure models
|
|
azure_response = completion(
|
|
model="azure/gpt-4o",
|
|
messages=[{"role": "user", "content": "Hello Azure!"}],
|
|
base_url=base_url
|
|
)
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
---
|
|
|
|
## Adding Custom Headers
|
|
|
|
Add Bifrost-specific headers for governance and tracking:
|
|
|
|
<Tabs group="litellm-sdk">
|
|
<Tab title="Python">
|
|
|
|
```python
|
|
from litellm import completion
|
|
|
|
# Add custom headers for Bifrost features
|
|
response = completion(
|
|
model="gpt-4o-mini",
|
|
messages=[{"role": "user", "content": "Hello!"}],
|
|
base_url="http://localhost:8080/litellm",
|
|
extra_headers={
|
|
"x-bf-vk": "your-virtual-key", # Virtual key for governance
|
|
}
|
|
)
|
|
|
|
print(response.choices[0].message.content)
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
---
|
|
|
|
## Using Direct Keys
|
|
|
|
Pass API keys directly to bypass Bifrost's key management. You can pass any provider's API key since Bifrost only looks for `Authorization` or `x-api-key` headers. This requires the **Allow Direct API keys** option to be enabled in Bifrost configuration.
|
|
|
|
> **Learn more:** See [Key Management](../features/keys-management#direct-key-bypass) for enabling direct API key usage.
|
|
|
|
<Tabs group="litellm-sdk">
|
|
<Tab title="Python">
|
|
|
|
```python
|
|
from litellm import completion
|
|
|
|
# Using OpenAI key directly
|
|
openai_response = completion(
|
|
model="gpt-4o-mini",
|
|
messages=[{"role": "user", "content": "Hello GPT!"}],
|
|
base_url="http://localhost:8080/litellm",
|
|
extra_headers={
|
|
"Authorization": "Bearer sk-your-openai-key"
|
|
}
|
|
)
|
|
|
|
# Using Anthropic key for Claude models
|
|
anthropic_response = completion(
|
|
model="claude-3-sonnet-20240229",
|
|
messages=[{"role": "user", "content": "Hello Claude!"}],
|
|
base_url="http://localhost:8080/litellm",
|
|
extra_headers={
|
|
"x-api-key": "sk-ant-your-anthropic-key"
|
|
}
|
|
)
|
|
|
|
# Using Azure with direct Azure key
|
|
import os
|
|
|
|
deployment = os.getenv("AZURE_OPENAI_DEPLOYMENT", "my-azure-deployment")
|
|
model = f"azure/{deployment}"
|
|
|
|
azure_response = completion(
|
|
model=model,
|
|
messages=[{"role": "user", "content": "Hello from LiteLLM (Azure demo)!"}],
|
|
base_url="http://localhost:8080/litellm",
|
|
api_key=os.getenv("AZURE_API_KEY", "your-azure-api-key"),
|
|
deployment_id=os.getenv("AZURE_OPENAI_DEPLOYMENT", "gpt-4o-aug"),
|
|
max_tokens=100,
|
|
extra_headers={
|
|
"x-bf-azure-endpoint": "https://your-resource.openai.azure.com",
|
|
}
|
|
)
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
---
|
|
|
|
## Supported Features
|
|
|
|
The LiteLLM integration supports all features that are available in both the LiteLLM SDK and Bifrost core functionality. Your existing LiteLLM code works seamlessly with Bifrost's enterprise features. 😄
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
- **[Governance Features](../features/governance)** - Virtual keys and team management
|
|
- **[Semantic Caching](../features/semantic-caching)** - Intelligent response caching
|
|
- **[Configuration](../quickstart/README)** - Provider setup and API key management
|