beyhan/bifrost
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
Beyhan Oğur
2026-04-26 21:52:23 +03:00
commit 880f412e2c
2662 changed files with 866266 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
docs/deployment-guides/how-to/install-make.mdx Normal file
View File

@@ -0,0 +1,77 @@
---
title: "Install make command"
description: "This guide explains how to install make command."
icon: "compact-disc"
---
## Windows
### Option A: Chocolatey (easy)
```
# Run in an elevated PowerShell (Run as Administrator)
choco install make
# verify
make --version
```
### Option B: Scoop (no admin needed)
```
# In a normal PowerShell
Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
iwr get.scoop.sh -useb | iex
scoop install make
make --version
```
### Option C: MSYS2 (full Unix-like env)
```
# 1) Install MSYS2 from https://www.msys2.org/
# 2) In "MSYS2 MSYS" terminal:
pacman -Syu # then reopen terminal if asked
pacman -S make
make --version
```
<Note> Visual Studio’s nmake is a different tool (not GNU make). </Note>
## Ubuntu / Debian
```
sudo apt update
# Pulls in compilers and common build tools, including make
sudo apt install build-essential
# (or just) sudo apt install make
make --version
```
## macOS
### Option A: Xcode Command Line Tools (most common)
```
xcode-select --install # follow the prompt
make --version
```
This provides Apple’s/BSD-flavored make, which is fine for most projects.
### Option B: Homebrew (get GNU make ≥ 4.x as gmake)
```
# Install Homebrew if needed: https://brew.sh
brew install make
gmake --version
```
If a project specifically requires GNU make as make, you can use:
echo 'alias make="gmake"' >> ~/.zshrc && source ~/.zshrc
## Troubleshooting tips
- If make isn’t found, restart your terminal (or on Windows, open a new PowerShell) so your PATH updates.
- Run which make (where make on Windows) to confirm which binary you’re using.
- For Windows builds that depend on Unix tools (sed, grep, etc.), prefer MSYS2 or WSL for a smoother experience.

444
docs/deployment-guides/how-to/multinode.mdx Normal file
View File

@@ -0,0 +1,444 @@
---
title: "Multinode Deployment"
description: "Deploy multiple Bifrost nodes with shared configuration for high availability in OSS deployments"
icon: "layer-group"
---
## Overview
Running multiple Bifrost nodes provides high availability, load distribution, and fault tolerance for your AI gateway. This guide covers the recommended approach for deploying multiple Bifrost nodes in OSS deployments.
<Warning>
Running multiple OSS Bifrost nodes with a Postgres backend is not supported.
Here is the short technical explanation:
- Bifrost is designed to keep all critical information in memory, including provider configs, API keys, budgets, usage, and traffic distribution.
- Once a node is initialized, it does not read this information back from the database.
- In the Enterprise version, we use a slightly modified version of RAFT to synchronize this state in real time across nodes, while the database acts only as a dumb store.
- Based on our current view, OSS is sufficient for startups and medium-scale teams, and can easily handle around 3,000–5,000 RPS on a single instance.
- If you need high availability and enterprise capabilities such as real-time synchronization, the Enterprise plan is the right fit.
- And yes, that is part of how we draw the OSS vs Enterprise line 💰.
</Warning>
### OSS vs Enterprise
| Aspect | OSS Approach | Enterprise Approach |
|--------|--------------|---------------------|
| **Configuration Source** | Shared `config.json` file | Database with P2P sync |
| **Sync Mechanism** | File sharing (ConfigMap, volumes) | Gossip protocol (real-time) |
| **Config Updates** | Modify file + restart nodes | UI/API with automatic propagation |
---
## How It Works
All configuration in Bifrost is loaded into memory at startup. For OSS multinode deployments, the recommended approach is to use `config.json` **without** `config_store` enabled.
### `config.json` as Single Source of Truth
When you deploy without `config_store`:
- **No database involved** - `config.json` is the only configuration source
- **Shared file** - All nodes read from the same `config.json` file
- **Identical configuration** - Since the source is shared, all nodes automatically have the same configuration
- **No sync needed** - The shared file itself ensures consistency
<Frame>
<img src="/media/oss-multinode.png" alt="OSS multi-node setup" />
</Frame>
---
## Why not to use `config_store` for Multinode OSS?
Using `config_store` (database-backed configuration) with multiple nodes in OSS creates a **synchronization problem**:
1. **Config changes are local** - When you update configuration via the UI or API, it updates the database and the in-memory config on that specific node only
2. **No propagation mechanism** - Other nodes don't know about the change; they keep their existing in-memory configuration
3. **Nodes become out of sync** - Different nodes end up with different configurations
4. **Restart required** - You'd have to restart all nodes after every config change to bring them back in sync
This defeats the purpose of having database-backed configuration with real-time updates.
<Warning>
Without P2P clustering (Enterprise feature), there's no mechanism to notify other nodes of configuration changes. For OSS multinode deployments, use the shared `config.json` approach instead.
</Warning>
### Enterprise Solution
Bifrost Enterprise includes **P2P clustering** with gossip protocol that automatically syncs configuration changes across all nodes in real-time. See the [Clustering documentation](/enterprise/clustering) for details.
---
## Setting Up Multinode OSS Deployment
### Example config.json
Create a `config.json` **without** `config_store` or `logs_store`:
<Note>
If you use PostgreSQL for `logs_store`, ensure the target database is UTF8 encoded. See [PostgreSQL UTF8 Requirement](../../quickstart/gateway/setting-up#postgresql-utf8-requirement).
</Note>
```json
{
"$schema": "https://www.getbifrost.ai/schema",
"client": {
"drop_excess_requests": false,
"enable_logging": false
},
"config_store": {
"enabled": false
},
"logs_store": {
"enabled": true,
"type": "postgres",
"config": {...}
},
"providers": {
"openai": {
"keys": [
{
"name": "openai-primary",
"value": "env.OPENAI_API_KEY",
"models": ["gpt-4o", "gpt-4o-mini"],
"weight": 1.0
}
]
},
"anthropic": {
"keys": [
{
"name": "anthropic-primary",
"value": "env.ANTHROPIC_API_KEY",
"models": ["claude-sonnet-4-20250514", "claude-3-5-haiku-20241022"],
"weight": 1.0
}
]
}
}
}
```
<Note>
Notice `config_store` is disabled. This ensures all configuration comes from the file only.
</Note>
### Kubernetes Deployment
Use a ConfigMap to share the same configuration across all pods:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: bifrost-config
namespace: default
data:
config.json: |
{
"$schema": "https://www.getbifrost.ai/schema",
"client": {
"drop_excess_requests": false,
"enable_logging": false
},
"config_store": {
"enabled": false
},
"logs_store": {
"enabled": true,
"type": "postgres",
"config": {...}
},
"providers": {
"openai": {
"keys": [
{
"name": "openai-primary",
"value": "env.OPENAI_API_KEY",
"models": ["gpt-4o", "gpt-4o-mini"],
"weight": 1.0
}
]
}
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: bifrost
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: bifrost
template:
metadata:
labels:
app: bifrost
spec:
containers:
- name: bifrost
image: maximhq/bifrost:latest
ports:
- containerPort: 8080
name: http
env:
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
name: provider-secrets
key: openai-api-key
volumeMounts:
- name: config
mountPath: /app
readOnly: true
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 1000m
memory: 1Gi
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
volumes:
- name: config
configMap:
name: bifrost-config
---
apiVersion: v1
kind: Service
metadata:
name: bifrost
namespace: default
spec:
type: LoadBalancer
selector:
app: bifrost
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
```
### Docker Compose
Share the configuration using a bind mount:
```yaml
version: '3.8'
services:
nginx:
image: nginx:alpine
ports:
- "80:80"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
- bifrost-1
- bifrost-2
- bifrost-3
bifrost-1:
image: maximhq/bifrost:latest
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
volumes:
- ./config.json:/app/config.json:ro
expose:
- "8080"
bifrost-2:
image: maximhq/bifrost:latest
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
volumes:
- ./config.json:/app/config.json:ro
expose:
- "8080"
bifrost-3:
image: maximhq/bifrost:latest
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
volumes:
- ./config.json:/app/config.json:ro
expose:
- "8080"
```
**nginx.conf** for load balancing:
```nginx
events {
worker_connections 1024;
}
http {
upstream bifrost {
least_conn;
server bifrost-1:8080;
server bifrost-2:8080;
server bifrost-3:8080;
}
server {
listen 80;
location / {
proxy_pass http://bifrost;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
}
location /health {
access_log off;
return 200 "healthy\n";
}
}
}
```
### Bare Metal / VM Deployment
For bare metal or VM deployments, distribute the configuration file using:
- **NFS mount** - Mount a shared NFS directory containing `config.json`
- **rsync** - Sync the config file from a central location to all nodes
- **Configuration management** - Use Ansible, Chef, or Puppet to deploy identical configs
Example with rsync:
```bash
# On config server - push to all nodes
for node in node1 node2 node3; do
rsync -avz /etc/bifrost/config.json $node:/etc/bifrost/config.json
done
# Restart nodes after config update
for node in node1 node2 node3; do
ssh $node "systemctl restart bifrost"
done
```
---
## Updating Configuration
To update configuration in a multinode OSS deployment:
1. **Modify the shared `config.json` file**
- Update the ConfigMap (Kubernetes)
- Edit the shared file (Docker Compose / bare metal)
2. **Restart the nodes**
- Rolling restart is supported - nodes can be restarted one at a time
- Each node picks up the new configuration on startup
### Kubernetes Rolling Restart
```bash
# Update ConfigMap
kubectl apply -f configmap.yaml
# Trigger rolling restart
kubectl rollout restart deployment/bifrost
# Watch the rollout
kubectl rollout status deployment/bifrost
```
### Docker Compose Restart
```bash
# After updating config.json
docker-compose restart bifrost-1
docker-compose restart bifrost-2
docker-compose restart bifrost-3
```
---
## Best Practices
### Use Environment Variables for Secrets
Never put API keys directly in `config.json`. Use the `env.` prefix to reference environment variables:
```json
{
"providers": {
"openai": {
"keys": [
{
"value": "env.OPENAI_API_KEY"
}
]
}
}
}
```
Then provide the actual keys via environment variables or Kubernetes secrets.
### Load Balancer Configuration
Always put a load balancer in front of your Bifrost nodes:
- **Kubernetes**: Use a Service with `type: LoadBalancer` or an Ingress
- **Docker/VMs**: Use nginx, HAProxy, or a cloud load balancer
### Health Checks
Configure health checks to ensure traffic only goes to healthy nodes:
- **Liveness endpoint**: `GET /health`
- **Readiness endpoint**: `GET /health`
### Resource Allocation
For production deployments:
```yaml
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 2Gi
```
---
## Summary
| Scenario | Recommendation |
|----------|----------------|
| Single node | Use `config_store` for UI access |
| Multinode OSS | Use shared `config.json` without `config_store` |
| Multinode Enterprise | Use P2P clustering with `config_store` |
For OSS multinode deployments, the shared `config.json` approach provides a simple, reliable way to keep all nodes in sync without the complexity of database synchronization.

185
docs/deployment-guides/how-to/nginx-reverse-proxy.mdx Normal file
View File

@@ -0,0 +1,185 @@
---
title: "Nginx reverse proxy"
description: "Run Bifrost behind NGINX with streaming-safe settings for SSE and WebSocket traffic"
icon: "shuffle"
---
This guide shows how to put NGINX in front of Bifrost for TLS termination, centralized routing, and load balancing.
<Note>
Incoming reverse-proxy behavior is configured in your infrastructure layer (NGINX/Ingress), not in `config.json`.
</Note>
---
## When to use this setup
- You want HTTPS termination in front of Bifrost.
- You run multiple Bifrost replicas and want L7 load balancing.
- You need one stable gateway URL for SDKs and agent clients.
---
## Docker Compose deployment
Use this when Bifrost and NGINX run as services in the same Compose project.
```yaml
services:
nginx:
image: nginx:alpine
ports:
- "80:80"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
- bifrost-1
- bifrost-2
- bifrost-3
bifrost-1:
image: maximhq/bifrost:latest
expose:
- "8080"
bifrost-2:
image: maximhq/bifrost:latest
expose:
- "8080"
bifrost-3:
image: maximhq/bifrost:latest
expose:
- "8080"
```
```nginx
events {
worker_connections 1024;
}
http {
upstream bifrost_backend {
least_conn;
server bifrost-1:8080;
server bifrost-2:8080;
server bifrost-3:8080;
}
server {
listen 80;
location / {
proxy_pass http://bifrost_backend;
# Preserve original request context
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Keep streaming responses stable
proxy_http_version 1.1;
proxy_buffering off;
proxy_request_buffering off;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
}
}
```
If you expose WebSocket traffic through the same endpoint, add upgrade headers in the same `location /` block:
```nginx
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
```
---
## VM or bare-metal deployment
Use the same NGINX `location /` settings as above, and point `upstream` servers to hostnames/IPs reachable from that VM.
If you terminate TLS directly on NGINX, add:
```nginx
listen 443 ssl;
server_name bifrost.example.com;
ssl_certificate /etc/nginx/certs/fullchain.pem;
ssl_certificate_key /etc/nginx/certs/privkey.pem;
```
---
## Kubernetes (NGINX Ingress)
If you deploy with Helm, use Ingress values instead of a standalone NGINX config:
```yaml
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
hosts:
- host: bifrost.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: bifrost-tls
hosts:
- bifrost.example.com
```
---
## Verify the proxy path
```bash
# Docker Compose: render final config and validate syntax
docker compose config
# Kubernetes: validate ingress manifest locally
kubectl apply --dry-run=client -f ingress.yaml
```
```bash
# Health check through reverse proxy
curl -i http://bifrost.example.com/health
# Streaming check through NGINX
curl -N http://bifrost.example.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini",
"stream": true,
"messages": [{"role": "user", "content": "test stream"}]
}'
```
If streaming responses arrive in delayed bursts, confirm buffering is disabled in NGINX or Ingress annotations.
---
## Related guides
- [Helm quick start](/deployment-guides/helm)
- [Helm values reference](/deployment-guides/helm/values)
- [Multinode deployment](/deployment-guides/how-to/multinode)
---
## Runnable example files
Use the complete Docker Compose + Helm/Kubernetes example in the repository:
- [docker-compose.yml](https://github.com/maximhq/bifrost/blob/main/examples/configs/withnginxreverseproxy/docker-compose.yml)
- [helm-values.yaml](https://github.com/maximhq/bifrost/blob/main/examples/configs/withnginxreverseproxy/helm-values.yaml)
- [k8s-ingress.yaml](https://github.com/maximhq/bifrost/blob/main/examples/configs/withnginxreverseproxy/k8s-ingress.yaml)