333 lines
8.2 KiB
Plaintext
333 lines
8.2 KiB
Plaintext
---
|
|
title: "What is an integration?"
|
|
description: "Protocol adapters that translate between Bifrost's unified API and provider-specific API formats like OpenAI, Anthropic, and Google GenAI."
|
|
icon: "box"
|
|
---
|
|
|
|
## Overview
|
|
|
|
An integration is a protocol adapter that translates between Bifrost's unified API and provider-specific API formats. Each integration handles request transformation, response normalization, and error mapping between the external API contract and Bifrost's internal processing pipeline.
|
|
|
|
Integrations enable you to utilize Bifrost's features like governance, MCP tools, load balancing, semantic caching, multi-provider support, and more, all while preserving your existing SDK-based architecture. Bifrost handles all the overhead of structure conversion, requiring only a single URL change to switch from direct provider APIs to Bifrost's gateway.
|
|
|
|
Bifrost converts the request/response format of the provider API to the Bifrost API format based on the integration used, so you don't have to.
|
|
|
|
---
|
|
|
|
## Quick Migration
|
|
|
|
### **Before (Direct Provider)**
|
|
|
|
```python
|
|
import openai
|
|
|
|
client = openai.OpenAI(
|
|
api_key="your-openai-key"
|
|
)
|
|
```
|
|
|
|
### **After (Bifrost)**
|
|
|
|
```python {4}
|
|
import openai
|
|
|
|
client = openai.OpenAI(
|
|
base_url="http://localhost:8080/openai", # Point to Bifrost
|
|
api_key="dummy-key" # Keys are handled in Bifrost now
|
|
)
|
|
```
|
|
|
|
**That's it!** Your application now benefits from Bifrost's features with no other changes.
|
|
|
|
---
|
|
|
|
## Supported Integrations
|
|
|
|
1. [OpenAI](./openai-sdk)
|
|
2. [Anthropic](./anthropic-sdk)
|
|
3. [Google GenAI](./genai-sdk)
|
|
4. [LiteLLM](./litellm-sdk)
|
|
5. [Langchain](./langchain-sdk)
|
|
6. [AWS Bedrock](./bedrock-sdk)
|
|
|
|
---
|
|
|
|
## Provider-Prefixed Models
|
|
|
|
Use multiple providers seamlessly by prefixing model names with the provider:
|
|
|
|
<Tabs>
|
|
<Tab title="OpenAI">
|
|
```python
|
|
import openai
|
|
|
|
# Single client, multiple providers
|
|
client = openai.OpenAI(
|
|
base_url="http://localhost:8080/openai",
|
|
api_key="dummy" # API keys configured in Bifrost
|
|
)
|
|
|
|
# OpenAI models
|
|
response1 = client.chat.completions.create(
|
|
model="gpt-4o-mini", # (default OpenAI since it's OpenAI's SDK)
|
|
messages=[{"role": "user", "content": "Hello!"}]
|
|
)
|
|
```
|
|
</Tab>
|
|
<Tab title="Anthropic">
|
|
```python
|
|
import openai
|
|
|
|
# Anthropic models using OpenAI SDK format
|
|
response2 = client.chat.completions.create(
|
|
model="anthropic/claude-3-sonnet-20240229",
|
|
messages=[{"role": "user", "content": "Hello!"}]
|
|
)
|
|
```
|
|
</Tab>
|
|
<Tab title="Azure">
|
|
```python
|
|
import openai
|
|
|
|
# Azure models
|
|
response4 = client.chat.completions.create(
|
|
model="azure/gpt-4o",
|
|
messages=[{"role": "user", "content": "Hello!"}]
|
|
)
|
|
```
|
|
</Tab>
|
|
<Tab title="Vertex">
|
|
```python
|
|
import openai
|
|
|
|
# Google Vertex models
|
|
response3 = client.chat.completions.create(
|
|
model="vertex/gemini-pro",
|
|
messages=[{"role": "user", "content": "Hello!"}]
|
|
)
|
|
```
|
|
</Tab>
|
|
<Tab title="Ollama">
|
|
```python
|
|
import openai
|
|
|
|
# Local Ollama models
|
|
response5 = client.chat.completions.create(
|
|
model="ollama/llama3.1:8b",
|
|
messages=[{"role": "user", "content": "Hello!"}]
|
|
)
|
|
```
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
---
|
|
|
|
## Direct API Usage
|
|
|
|
For custom HTTP clients or when you have existing provider-specific setup and want to use Bifrost gateway without restructuring your codebase:
|
|
|
|
```python {5,18,31,}
|
|
import requests
|
|
|
|
# Fully OpenAI compatible endpoint
|
|
response = requests.post(
|
|
"http://localhost:8080/openai/v1/chat/completions",
|
|
headers={
|
|
"Authorization": f"Bearer {openai_key}",
|
|
"Content-Type": "application/json"
|
|
},
|
|
json={
|
|
"model": "gpt-4o-mini",
|
|
"messages": [{"role": "user", "content": "Hello!"}]
|
|
}
|
|
)
|
|
|
|
# Fully Anthropic compatible endpoint
|
|
response = requests.post(
|
|
"http://localhost:8080/anthropic/v1/messages",
|
|
headers={
|
|
"Content-Type": "application/json",
|
|
},
|
|
json={
|
|
"model": "claude-3-sonnet-20240229",
|
|
"max_tokens": 1000,
|
|
"messages": [{"role": "user", "content": "Hello!"}]
|
|
}
|
|
)
|
|
|
|
# Fully Google GenAI compatible endpoint
|
|
response = requests.post(
|
|
"http://localhost:8080/genai/v1beta/models/gemini-1.5-flash/generateContent",
|
|
headers={
|
|
"Content-Type": "application/json",
|
|
},
|
|
json={
|
|
"contents": [
|
|
{"parts": [{"text": "Hello!"}]}
|
|
],
|
|
"generation_config": {
|
|
"max_output_tokens": 1000,
|
|
"temperature": 1
|
|
}
|
|
}
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## Listing Models
|
|
|
|
All integrations support listing available models through their respective list models endpoints (e.g., `/openai/v1/models`, `/anthropic/v1/models`). By default, list models requests return models from **all configured providers** in Bifrost.
|
|
|
|
### Filtering by Provider
|
|
|
|
You can control which provider's models to list using the `x-bf-list-models-provider` header:
|
|
|
|
<Tabs>
|
|
<Tab title="Python">
|
|
|
|
```python
|
|
import openai
|
|
|
|
client = openai.OpenAI(
|
|
base_url="http://localhost:8080/openai",
|
|
api_key="dummy-key"
|
|
)
|
|
|
|
# List models from all providers (default behavior)
|
|
all_models = client.models.list()
|
|
|
|
# List models from a specific provider only
|
|
openai_models = client.models.list(
|
|
extra_headers={
|
|
"x-bf-list-models-provider": "openai"
|
|
}
|
|
)
|
|
|
|
anthropic_models = client.models.list(
|
|
extra_headers={
|
|
"x-bf-list-models-provider": "anthropic"
|
|
}
|
|
)
|
|
```
|
|
|
|
</Tab>
|
|
<Tab title="JavaScript">
|
|
|
|
```javascript
|
|
import OpenAI from "openai";
|
|
|
|
const openai = new OpenAI({
|
|
baseURL: "http://localhost:8080/openai",
|
|
apiKey: "dummy-key",
|
|
});
|
|
|
|
// List models from all providers (default behavior)
|
|
const allModels = await openai.models.list();
|
|
|
|
// List models from a specific provider only
|
|
const openaiModels = await openai.models.list({
|
|
headers: {
|
|
"x-bf-list-models-provider": "openai",
|
|
},
|
|
});
|
|
|
|
const anthropicModels = await openai.models.list({
|
|
headers: {
|
|
"x-bf-list-models-provider": "anthropic",
|
|
},
|
|
});
|
|
```
|
|
|
|
</Tab>
|
|
<Tab title="cURL">
|
|
|
|
```bash
|
|
# List models from all providers (default)
|
|
curl http://localhost:8080/openai/v1/models
|
|
|
|
# List models from specific provider
|
|
curl http://localhost:8080/openai/v1/models \
|
|
-H "x-bf-list-models-provider: openai"
|
|
|
|
# Explicitly request all providers
|
|
curl http://localhost:8080/openai/v1/models \
|
|
-H "x-bf-list-models-provider: all"
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
### Header Behavior
|
|
|
|
| Header Value | Behavior |
|
|
|--------------|----------|
|
|
| Not set (default) | Lists models from **all configured providers** |
|
|
| `all` | Lists models from **all configured providers** |
|
|
| `openai` | Lists models from **OpenAI provider only** |
|
|
| `anthropic` | Lists models from **Anthropic provider only** |
|
|
| `vertex` | Lists models from **Vertex AI provider only** |
|
|
| Any valid provider | Lists models from that specific provider |
|
|
|
|
### Response Fields
|
|
|
|
When listing models from all providers, some provider-specific fields may be empty or contain default values if the information is not available from all providers. This is normal behavior as different providers expose different model metadata.
|
|
|
|
---
|
|
|
|
|
|
## Migration Strategies
|
|
|
|
### **Gradual Migration**
|
|
|
|
1. **Start with development** - Test Bifrost in dev environment
|
|
2. **Canary deployment** - Route 5% of traffic through Bifrost
|
|
3. **Feature-by-feature** - Migrate specific endpoints gradually
|
|
4. **Full migration** - Switch all traffic to Bifrost
|
|
|
|
### **Blue-Green Migration**
|
|
|
|
```python
|
|
import os
|
|
import random
|
|
|
|
# Route traffic based on feature flag
|
|
def get_base_url(provider: str) -> str:
|
|
if os.getenv("USE_BIFROST", "false") == "true":
|
|
return f"http://bifrost:8080/{provider}"
|
|
else:
|
|
return f"https://api.{provider}.com"
|
|
|
|
# Gradual rollout
|
|
def should_use_bifrost() -> bool:
|
|
rollout_percentage = int(os.getenv("BIFROST_ROLLOUT", "0"))
|
|
return random.randint(1, 100) <= rollout_percentage
|
|
```
|
|
|
|
### **Feature Flag Integration**
|
|
|
|
```python
|
|
# Using feature flags for safe migration
|
|
import openai
|
|
from feature_flags import get_flag
|
|
|
|
def create_client():
|
|
if get_flag("use_bifrost_openai"):
|
|
base_url = "http://bifrost:8080/openai"
|
|
else:
|
|
base_url = "https://api.openai.com"
|
|
|
|
return openai.OpenAI(
|
|
base_url=base_url,
|
|
api_key=os.getenv("OPENAI_API_KEY")
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
- **[HTTP Transport Overview](../quickstart/gateway/setting-up)** - Main HTTP transport guide
|
|
- **[Endpoints](../openapi/openapi.json)** - Complete API reference
|
|
- **[Configuration](../quickstart/gateway/provider-configuration)** - Provider setup and config
|