beyhan/bifrost
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
Beyhan Oğur
2026-04-26 21:52:23 +03:00
commit 880f412e2c
2662 changed files with 866266 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

978
docs/features/observability/otel.mdx Normal file
View File

@@ -0,0 +1,978 @@
---
title: "OpenTelemetry (OTel)"
description: "Integrate with OpenTelemetry collectors for enterprise observability and distributed tracing"
icon: "bolt"
---
## Overview
<Frame>
<img src="/media/grafana-otel-traces.png" alt="Okta Applications page" />
</Frame>
The **OTel plugin** enables seamless integration with OpenTelemetry Protocol (OTLP) collectors, allowing you to send LLM traces to your existing observability infrastructure. Connect Bifrost to platforms like Grafana Cloud, Datadog, New Relic, Honeycomb, or self-hosted collectors.
All traces follow OpenTelemetry semantic conventions, making it easy to correlate LLM operations with your broader application telemetry.
---
## Supported Trace Formats
The plugin supports multiple trace formats to match your observability platform:
| Format | Description | Use Case | Status |
|--------|-------------|----------|----------|
| `genai_extension` | OpenTelemetry GenAI semantic conventions | **Recommended** - Standard OTel format with rich LLM metadata | ✅ Released |
| `vercel` | Vercel AI SDK format | For Vercel AI SDK compatibility | 🔄 Coming soon |
| `open_inference` | Arize OpenInference format | For Arize Phoenix and OpenInference tools | 🔄 Coming soon |
---
## Configuration
### Required Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `service_name` | `string` | ❌ No | Service name to be used for tracing, defaults to `bifrost` |
| `collector_url` | `string` | ✅ Yes | OTLP collector endpoint URL |
| `trace_type` | `string` | ✅ Yes | One of: `genai_extension`, `vercel`, `open_inference` |
| `protocol` | `string` | ✅ Yes | Transport protocol: `http` or `grpc` |
| `headers` | `object` | ❌ No | Custom headers for authentication (supports `env.VAR_NAME`) |
| `tls_ca_cert` | `string` | ❌ No | File path to client CA certificate for TLS. Optional. Works with both gRPC and HTTP protocol |
### Environment Variable Substitution
Headers support environment variable substitution using the `env.` prefix:
```json
{
"headers": {
"Authorization": "env.OTEL_API_KEY",
"X-Custom-Header": "env.CUSTOM_VALUE"
}
}
```
### Resource Attributes
The plugin supports the standard `OTEL_RESOURCE_ATTRIBUTES` environment variable. Any attributes defined in this variable will be automatically attached to every span emitted by the plugin.
```bash
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,service.version=1.2.3,team.name=platform"
```
These attributes appear as resource-level metadata on all traces:
```json
{
"resource": {
"attributes": {
"service.name": "bifrost",
"deployment.environment": "production",
"service.version": "1.2.3",
"team.name": "platform"
}
}
}
```
This is useful for:
- **Environment identification** - Distinguish between production, staging, and development traces
- **Service versioning** - Track which version of your service generated the trace
- **Team attribution** - Tag traces with team ownership for filtering and alerting
- **Custom metadata** - Add any key-value pairs relevant to your observability needs
---
## Setup
<Tabs group="setup-method">
<Tab title="UI">
![Otel UI setup](../../media/otel-ui-setup.png)
</Tab>
<Tab title="Go SDK">
```go
package main
import (
"context"
bifrost "github.com/maximhq/bifrost/core"
"github.com/maximhq/bifrost/core/schemas"
"github.com/maximhq/bifrost/framework/pricing"
otel "github.com/maximhq/bifrost/plugins/otel"
)
func main() {
ctx := context.Background()
logger := schemas.NewLogger()
// Initialize pricing manager (required for cost calculation)
pricingManager := pricing.NewPricingManager(logger)
// Initialize OTel plugin
otelPlugin, err := otel.Init(ctx, &otel.Config{
ServiceName: "bifrost",
CollectorURL: "http://localhost:4318",
TraceType: otel.TraceTypeGenAIExtension,
Protocol: otel.ProtocolHTTP,
Headers: map[string]string{
"Authorization": "env.OTEL_API_KEY",
},
}, logger, pricingManager)
if err != nil {
panic(err)
}
// Initialize Bifrost with the plugin
client, err := bifrost.Init(ctx, schemas.BifrostConfig{
Account: &yourAccount,
LLMPlugins: []schemas.LLMPlugin{otelPlugin},
})
if err != nil {
panic(err)
}
defer client.Shutdown()
// All requests are now traced to OTel collector
}
```
</Tab>
<Tab title="config.json">
For Gateway mode, configure via `config.json`:
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "http://localhost:4318",
"trace_type": "genai_extension",
"protocol": "http",
"headers": {
"Authorization": "env.OTEL_API_KEY"
}
}
}
]
}
```
If you need to connect to an OTEL collector that requires TLS, configure `tls_ca_cert`:
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "localhost:4317",
"trace_type": "genai_extension",
"protocol": "grpc",
"tls_ca_cert": "/path/to/your/ca.cert",
"headers": {
"Authorization": "env.OTEL_API_KEY"
}
}
}
]
}
```
</Tab>
</Tabs>
---
## Quick Start with Docker
Get started quickly with a complete observability stack using the included Docker Compose configuration:
```yml
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
container_name: otel-collector
command: ["--config=/etc/otelcol/config.yaml"]
configs:
- source: otel-collector-config
target: /etc/otelcol/config.yaml
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "8888:8888" # Collector /metrics
- "9464:9464" # Prometheus scrape endpoint
- "13133:13133" # Health check
- "1777:1777" # pprof
- "55679:55679" # zpages
restart: unless-stopped
depends_on:
- tempo
tempo:
image: grafana/tempo:latest
container_name: tempo
command: [ "-config.file=/etc/tempo.yaml" ]
configs:
- source: tempo-config
target: /etc/tempo.yaml
ports:
- "3200:3200" # tempo HTTP API
expose:
- "4317" # OTLP gRPC (internal)
volumes:
- tempo-data:/var/tempo
restart: unless-stopped
prometheus:
image: prom/prometheus:latest
container_name: prometheus
depends_on:
- otel-collector
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--web.console.libraries=/usr/share/prometheus/console_libraries"
- "--web.console.templates=/usr/share/prometheus/consoles"
- "--web.enable-remote-write-receiver"
ports:
- "9090:9090"
volumes:
- prometheus-data:/prometheus
configs:
- source: prometheus-config
target: /etc/prometheus/prometheus.yml
restart: unless-stopped
grafana:
image: grafana/grafana:latest
container_name: grafana
depends_on:
- prometheus
- tempo
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: "grafana-pyroscope-app,grafana-exploretraces-app,grafana-metricsdrilldown-app"
GF_PLUGINS_ENABLE_ALPHA: "true"
GF_INSTALL_PLUGINS: ""
ports:
- "4000:3000"
volumes:
- grafana-data:/var/lib/grafana
configs:
- source: grafana-datasources
target: /etc/grafana/provisioning/datasources/datasources.yml
restart: unless-stopped
configs:
otel-collector-config:
content: |
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
exporters:
prometheus:
endpoint: 0.0.0.0:9464
namespace: otel
const_labels:
source: otelcol
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
debug:
verbosity: detailed
extensions:
health_check:
endpoint: 0.0.0.0:13133
pprof:
endpoint: 0.0.0.0:1777
zpages:
endpoint: 0.0.0.0:55679
service:
extensions: [health_check, pprof, zpages]
telemetry:
logs:
level: debug
metrics:
level: detailed
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [debug, otlp/tempo]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [debug, prometheus]
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug]
tempo-config:
content: |
server:
http_listen_port: 3200
log_level: info
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
ingester:
max_block_duration: 5m
trace_idle_period: 10s
compactor:
compaction:
block_retention: 1h
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks
metrics_generator:
registry:
external_labels:
source: tempo
storage:
path: /var/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
prometheus-config:
content: |
global:
scrape_interval: 15s
scrape_configs:
- job_name: "otelcol-internal"
static_configs:
- targets: ["otel-collector:8888"]
- job_name: "otelcol-exporter"
static_configs:
- targets: ["otel-collector:9464"]
- job_name: "tempo"
static_configs:
- targets: ["tempo:3200"]
grafana-datasources:
content: |
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
orgId: 1
url: http://prometheus:9090
isDefault: true
editable: true
- name: Tempo
uid: tempo
type: tempo
access: proxy
orgId: 1
url: http://tempo:3200
editable: true
jsonData:
tracesToMetrics:
datasourceUid: prometheus
nodeGraph:
enabled: true
volumes:
prometheus-data:
grafana-data:
tempo-data:
```
This launches:
- **OTel Collector** - Receives traces on ports 4317 (gRPC) and 4318 (HTTP)
- **Tempo** - Distributed tracing backend
- **Prometheus** - Metrics collection
- **Grafana** - Visualization dashboard
Access Grafana at `http://localhost:3000` (default credentials: admin/admin)
<Frame>
<img src="/media/grafana-otel-traces.png" alt="Okta Applications page" />
</Frame>
---
## Popular Platform Integrations
<Tabs group="platforms">
<Tab title="Grafana Cloud">
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "https://otlp-gateway-prod-us-central-0.grafana.net/otlp",
"trace_type": "genai_extension",
"protocol": "http",
"headers": {
"Authorization": "env.GRAFANA_CLOUD_API_KEY"
}
}
}
]
}
```
Set environment variable:
```bash
export GRAFANA_CLOUD_API_KEY="Basic <your-base64-encoded-token>"
```
</Tab>
<Tab title="Datadog">
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "https://trace.agent.datadoghq.com",
"trace_type": "genai_extension",
"protocol": "http",
"headers": {
"DD-API-KEY": "env.DATADOG_API_KEY"
}
}
}
]
}
```
Set environment variable:
```bash
export DATADOG_API_KEY="your-datadog-api-key"
```
</Tab>
<Tab title="New Relic">
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "https://otlp.nr-data.net:4318",
"trace_type": "genai_extension",
"protocol": "http",
"headers": {
"api-key": "env.NEW_RELIC_LICENSE_KEY"
}
}
}
]
}
```
Set environment variable:
```bash
export NEW_RELIC_LICENSE_KEY="your-license-key"
```
</Tab>
<Tab title="Honeycomb">
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "https://api.honeycomb.io",
"trace_type": "genai_extension",
"protocol": "http",
"headers": {
"x-honeycomb-team": "env.HONEYCOMB_API_KEY",
"x-honeycomb-dataset": "bifrost-traces"
}
}
}
]
}
```
Set environment variable:
```bash
export HONEYCOMB_API_KEY="your-api-key"
```
</Tab>
<Tab title="Langfuse">
[Langfuse](https://langfuse.com) is an open-source LLM observability platform that accepts OpenTelemetry traces via its OTLP endpoint.
<Tabs>
<Tab title="UI">
Configure the OTel plugin with the following settings:
| Field | Value |
|-------|-------|
| **Collector URL** | `https://cloud.langfuse.com/api/public/otel` (EU) or `https://us.cloud.langfuse.com/api/public/otel` (US) |
| **Trace Type** | `genai_extension` |
| **Protocol** | `http` (required - Langfuse does not support gRPC) |
| **Headers** | `Authorization`: `env.LANGFUSE_AUTH` |
</Tab>
<Tab title="config.json">
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "https://cloud.langfuse.com/api/public/otel",
"trace_type": "genai_extension",
"protocol": "http",
"headers": {
"Authorization": "env.LANGFUSE_AUTH"
}
}
}
]
}
```
For US region, use `https://us.cloud.langfuse.com/api/public/otel` instead.
</Tab>
</Tabs>
Set up the environment variable with your Langfuse API keys:
```bash
# Generate base64 auth string from your Langfuse API keys
export LANGFUSE_AUTH="Basic $(echo -n 'pk-lf-xxx:sk-lf-xxx' | base64)"
```
Replace `pk-lf-xxx` and `sk-lf-xxx` with your Langfuse public and secret keys from your project settings.
<Note>
Langfuse only supports HTTP protocol. Do not use gRPC.
</Note>
See the [Langfuse OpenTelemetry documentation](https://langfuse.com/integrations/native/opentelemetry) for more details.
</Tab>
<Tab title="Self-Hosted">
Use the included Docker Compose stack or point to your own collector:
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "http://your-collector:4318",
"trace_type": "genai_extension",
"protocol": "http"
}
}
]
}
```
</Tab>
</Tabs>
---
## Captured Data
Each trace includes comprehensive LLM operation metadata following OpenTelemetry semantic conventions:
### Span Attributes
- **Span Name**: Based on request type (`gen_ai.chat`, `gen_ai.text`, `gen_ai.embedding`, etc.)
- **Service Info**: `service.name=bifrost`, `service.version`
- **Provider & Model**: `gen_ai.provider.name`, `gen_ai.request.model`
### Request Parameters
- Temperature, max_tokens, top_p, stop sequences
- Presence/frequency penalties
- Tool configurations and parallel tool calls
- Custom parameters via `ExtraParams`
### Input/Output Data
- Complete chat history with role-based messages
- Prompt text for completions
- Response content with role attribution
- Tool calls and results
### Performance Metrics
- Token usage (prompt, completion, total)
- Cost calculations in dollars
- Latency and timing (start/end timestamps)
- Error details with status codes
### Example Span
```json
{
"name": "gen_ai.chat",
"attributes": {
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "gpt-4",
"gen_ai.request.temperature": 0.7,
"gen_ai.request.max_tokens": 1000,
"gen_ai.usage.prompt_tokens": 45,
"gen_ai.usage.completion_tokens": 128,
"gen_ai.usage.total_tokens": 173,
"gen_ai.usage.cost": 0.0052
}
}
```
<Frame>
<img src="/media/grafana-otel-span-details.png" alt="Okta Applications page" />
</Frame>
---
## Supported Request Types
The OTel plugin captures all Bifrost request types:
- **Chat Completion** (streaming and non-streaming) → `gen_ai.chat`
- **Text Completion** (streaming and non-streaming) → `gen_ai.text`
- **Embeddings** → `gen_ai.embedding`
- **Speech Generation** (streaming and non-streaming) → `gen_ai.speech`
- **Transcription** (streaming and non-streaming) → `gen_ai.transcription`
- **Responses API** → `gen_ai.responses`
---
## Protocol Support
### HTTP (OTLP/HTTP)
Uses HTTP/1.1 or HTTP/2 with JSON or Protobuf encoding:
```json
{
"collector_url": "http://localhost:4318",
"protocol": "http"
}
```
Default port: **4318**
### gRPC (OTLP/gRPC)
Uses gRPC with Protobuf encoding for lower latency:
```json
{
"collector_url": "localhost:4317",
"protocol": "grpc"
}
```
Default port: **4317**
---
## Metrics Push (Cluster Mode)
<Note>
**Multi-node deployments**: If you are running multiple Bifrost nodes, use push-based metrics for accurate aggregation. Pull-based `/metrics` scraping may miss nodes behind a load balancer.
</Note>
The OTel plugin supports **push-based metrics export** via OTLP, which is essential for multi-node cluster deployments. Instead of relying on Prometheus scraping each node's `/metrics` endpoint (which can miss nodes behind a load balancer), all nodes actively push metrics to a central OTEL Collector.
### Configuration
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `metrics_enabled` | `boolean` | ❌ No | Enable push-based metrics export (default: `false`) |
| `metrics_endpoint` | `string` | ✅ Yes (if enabled) | OTLP metrics endpoint URL |
| `metrics_push_interval` | `integer` | ❌ No | Push interval in seconds (default: `15`, range: 1-300) |
### Example Configuration
<Tabs group="metrics-config">
<Tab title="HTTP Protocol">
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "http://otel-collector:4318/v1/traces",
"trace_type": "genai_extension",
"protocol": "http",
"metrics_enabled": true,
"metrics_endpoint": "http://otel-collector:4318/v1/metrics",
"metrics_push_interval": 15
}
}
]
}
```
</Tab>
<Tab title="gRPC Protocol">
```json
{
"plugins": [
{
"enabled": true,
"name": "otel",
"config": {
"service_name": "bifrost",
"collector_url": "otel-collector:4317",
"trace_type": "genai_extension",
"protocol": "grpc",
"metrics_enabled": true,
"metrics_endpoint": "otel-collector:4317",
"metrics_push_interval": 15
}
}
]
}
```
</Tab>
</Tabs>
### Pushed Metrics
These are the same **Prometheus-style metrics** from the telemetry plugin, pushed via OTLP protocol to a central collector:
| Metric | Type | Description |
|--------|------|-------------|
| `bifrost_upstream_requests_total` | Counter | Total requests to upstream providers |
| `bifrost_success_requests_total` | Counter | Successful upstream requests |
| `bifrost_error_requests_total` | Counter | Error requests with status code labels |
| `bifrost_input_tokens_total` | Counter | Total input tokens |
| `bifrost_output_tokens_total` | Counter | Total output tokens |
| `bifrost_cache_hits_total` | Counter | Cache hits |
| `bifrost_cost_total` | Counter | Total cost in USD |
| `bifrost_upstream_latency_seconds` | Histogram | Upstream request latency |
| `bifrost_stream_first_token_latency_seconds` | Histogram | Time to first token |
| `bifrost_stream_inter_token_latency_seconds` | Histogram | Inter-token latency |
| `http_requests_total` | Counter | Total HTTP requests |
| `http_request_duration_seconds` | Histogram | HTTP request duration |
### OTEL Collector Configuration
Configure your OTEL Collector to receive OTLP metrics and export to your preferred backend (Datadog, Prometheus, etc.):
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 10s
send_batch_size: 1000
exporters:
# For Datadog
datadog:
api:
key: ${DD_API_KEY}
# Or for Prometheus remote write
prometheusremotewrite:
endpoint: "http://prometheus:9090/api/v1/write"
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [datadog] # or prometheusremotewrite
```
### Why Push vs Pull?
| Aspect | Pull (`/metrics` scrape) | Push (OTEL metrics) |
|--------|--------------------------|---------------------|
| Load balancer | May miss nodes | All nodes push |
| Service discovery | Required | Not required |
| Scraper configuration | Per-node endpoints | Single collector |
| Cluster aggregation | Query-side `sum()` | Collector handles it |
For **single-node deployments**, pull-based `/metrics` scraping works well. For **multi-node clusters**, push-based metrics ensures all nodes are captured.
---
## Advanced Features
### Automatic Span Management
- Spans are tracked with a **20-minute TTL** using an efficient sync.Map implementation
- Automatic cleanup prevents memory leaks for long-running processes
- Handles streaming requests with accumulator for chunked responses
### Async Emission
All span emissions happen asynchronously in background goroutines:
```go
// Zero impact on request latency
go func() {
p.client.Emit(ctx, spans)
}()
```
### Streaming Support
The plugin accumulates streaming chunks and emits a single complete span when the stream finishes, providing accurate token counts and costs.
### Environment Variable Security
Sensitive credentials never appear in config files:
```json
{
"headers": {
"Authorization": "env.OTEL_API_KEY"
}
}
```
The plugin reads `OTEL_API_KEY` from the environment at runtime.
---
## When to Use
### OTel Plugin
Choose the OTel plugin when you:
- Have existing OpenTelemetry infrastructure
- Need to correlate LLM traces with application traces
- Require compliance with enterprise observability standards
- Want vendor flexibility (switch backends without code changes)
- Need multi-service distributed tracing
### vs. Built-in Observability
Use [Built-in Observability](./default) for:
- Local development and testing
- Simple self-hosted deployments
- No external dependencies
- Direct database access to logs
### vs. Maxim Plugin
Use the [Maxim Plugin](./maxim) for:
- Advanced LLM evaluation and testing
- Prompt engineering and experimentation
- Team collaboration and governance
- Production monitoring with alerts
- Dataset management and curation
---
## Troubleshooting
### Connection Issues
Verify collector is reachable:
```bash
# Test HTTP endpoint
curl -v http://localhost:4318/v1/traces
# Test gRPC endpoint (requires grpcurl)
grpcurl -plaintext localhost:4317 list
```
### Missing Traces
Check Bifrost logs for emission errors:
```bash
# Enable debug logging
bifrost-http --log-level debug
```
### Authentication Failures
Verify environment variables are set:
```bash
echo $OTEL_API_KEY
```
---
## Next Steps
- **[Built-in Observability](./default)** - Local logging for development
- **[Maxim Plugin](./maxim)** - Advanced LLM evaluation and monitoring
- **[Telemetry](../telemetry)** - Prometheus metrics and dashboards