413 lines
20 KiB
Plaintext
413 lines
20 KiB
Plaintext
---
|
|
title: "Model Catalog"
|
|
description: "A centralized system for managing model information, pricing, and capabilities across all supported AI providers."
|
|
icon: "book-open"
|
|
---
|
|
|
|
The Model Catalog is a foundational component of Bifrost that provides a unified interface for managing AI models, including their pricing, capabilities, and availability. It serves as a centralized repository for all model-related information, enabling dynamic cost calculation, intelligent model routing, and efficient resource management.
|
|
|
|
<Info>
|
|
**Related Documentation**: The Model Catalog powers Bifrost's intelligent routing system. See [Provider Routing](/providers/provider-routing) for detailed examples of how governance and load balancing use the catalog to make routing decisions, including cross-provider scenarios and weighted routing via proxy providers.
|
|
</Info>
|
|
|
|
## Core Features
|
|
|
|
### **1. Automatic Pricing Synchronization**
|
|
The Model Catalog manages pricing data through a two-phase approach:
|
|
|
|
**Startup Behavior:**
|
|
- **With ConfigStore**: Downloads a pricing sheet from Maxim's datasheet, persists it to the config store, and then loads it into memory for fast lookups.
|
|
- **Without ConfigStore**: Downloads the pricing sheet directly into memory on every startup.
|
|
|
|
**Ongoing Synchronization:**
|
|
- When ConfigStore is available, an automatic sync occurs every 24 hours to keep pricing data current.
|
|
- All pricing data is cached in memory for O(1) lookup performance during cost calculations.
|
|
|
|
This ensures that cost calculations always use the latest pricing information from AI providers while maintaining optimal performance.
|
|
|
|
### **2. Multi-Modal Cost Calculation**
|
|
It supports diverse pricing models across different AI operation types:
|
|
- **Text Operations**: Token-based pricing for chat completions, text completions, responses, and embeddings. Cache-read/cache-write pricing applies to chat/text/responses when providers surface prompt cache token details.
|
|
- **Audio Processing**: Character-based, token-based, and duration-based pricing for speech synthesis and transcription, with audio token detail breakdown. Speech responses populate `usage.input_chars` so speech can be billed by input characters in addition to tokens/duration.
|
|
- **Image Processing**: Per-image (`input_cost_per_image`/`output_cost_per_image`), per-pixel (`input_cost_per_pixel`/`output_cost_per_pixel`), or token-based pricing with text/image token breakdown.
|
|
- **Video Processing**: Token-based or duration-based pricing. Input can use prompt tokens or `input_cost_per_video_per_second`; output can use completion tokens or fall back to `output_cost_per_video_per_second` / `output_cost_per_second`.
|
|
- **Reranking**: Input/output token pricing with search query cost support.
|
|
- **Prompt Caching**: Separate rates for cache-read tokens (`cached_read_tokens`) and cache-creation tokens (`cached_write_tokens`), both surfaced under `prompt_tokens_details` (see [Prompt Cache Cost Calculation](#prompt-cache-cost-calculation)).
|
|
|
|
### **3. Model Information Management**
|
|
The Model Catalog maintains a pool of available models for each provider, populated from both pricing data and provider list models APIs. This enables:
|
|
- **Model Discovery**: Listing all available models for a given provider
|
|
- **Provider Discovery**: Finding all providers that support a specific model with intelligent cross-provider resolution (OpenRouter, Vertex, Groq, Bedrock)
|
|
- **Model Validation**: Checking if a model is allowed for a provider based on allowed models lists (supports provider-prefixed entries)
|
|
|
|
### **4. Intelligent Cache Cost Handling**
|
|
It integrates with semantic caching to provide accurate cost calculations:
|
|
- **Cache Hits**: Zero cost for direct cache hits, and embedding cost only for semantic matches.
|
|
- **Cache Misses**: Combined cost of the base model usage plus the embedding generation cost for cache storage.
|
|
|
|
### **5. Tiered Pricing Support**
|
|
The system automatically applies different pricing rates for high-token contexts, reflecting real provider pricing models. Two tiers are supported: above 128k tokens and above 200k tokens, with the higher tier taking precedence when both are configured.
|
|
|
|
## Configuration
|
|
|
|
The `ModelCatalog` can be configured during initialization by passing a `Config` struct.
|
|
|
|
```go
|
|
type Config struct {
|
|
PricingURL *string `json:"pricing_url,omitempty"`
|
|
PricingSyncInterval *time.Duration `json:"pricing_sync_interval,omitempty"`
|
|
}
|
|
```
|
|
|
|
- **`PricingURL`**: Overrides the default URL (`https://getbifrost.ai/datasheet`) for downloading the pricing sheet.
|
|
- **`PricingSyncInterval`**: Customizes the interval for periodic pricing data synchronization. The default is 24 hours.
|
|
|
|
This configuration is passed during the initialization of the `ModelCatalog`:
|
|
|
|
```go
|
|
config := &modelcatalog.Config{
|
|
PricingURL: "https://my-custom-url.com/pricing.json",
|
|
}
|
|
modelCatalog, err := modelcatalog.Init(context.Background(), config, configStore, logger)
|
|
```
|
|
|
|
## Architecture
|
|
|
|
### ModelCatalog
|
|
The `ModelCatalog` is the central component that handles all model and pricing operations:
|
|
|
|
```go
|
|
type ModelCatalog struct {
|
|
configStore configstore.ConfigStore
|
|
logger schemas.Logger
|
|
|
|
pricingURL string
|
|
pricingSyncInterval time.Duration
|
|
|
|
// In-memory cache for fast access
|
|
pricingData map[string]configstoreTables.TableModelPricing
|
|
mu sync.RWMutex
|
|
|
|
modelPool map[schemas.ModelProvider][]string
|
|
|
|
// Background sync worker
|
|
syncTicker *time.Ticker
|
|
done chan struct{}
|
|
wg sync.WaitGroup
|
|
syncCtx context.Context
|
|
syncCancel context.CancelFunc
|
|
}
|
|
```
|
|
|
|
### Pricing Data Structure
|
|
Each model's pricing information includes comprehensive cost metrics, supporting various modalities and tiered pricing:
|
|
|
|
```go
|
|
// PricingEntry represents a single model's pricing information.
|
|
// The fields below are an excerpt — see framework/modelcatalog/main.go for the full definition.
|
|
type PricingEntry struct {
|
|
BaseModel string `json:"base_model,omitempty"`
|
|
Provider string `json:"provider"`
|
|
Mode string `json:"mode"`
|
|
|
|
// Costs - Text
|
|
InputCostPerToken float64 `json:"input_cost_per_token"`
|
|
OutputCostPerToken float64 `json:"output_cost_per_token"`
|
|
InputCostPerTokenBatches *float64 `json:"input_cost_per_token_batches,omitempty"`
|
|
OutputCostPerTokenBatches *float64 `json:"output_cost_per_token_batches,omitempty"`
|
|
InputCostPerTokenPriority *float64 `json:"input_cost_per_token_priority,omitempty"`
|
|
OutputCostPerTokenPriority *float64 `json:"output_cost_per_token_priority,omitempty"`
|
|
InputCostPerTokenAbove200kTokens *float64 `json:"input_cost_per_token_above_200k_tokens,omitempty"`
|
|
OutputCostPerTokenAbove200kTokens *float64 `json:"output_cost_per_token_above_200k_tokens,omitempty"`
|
|
|
|
// Costs - Cache
|
|
CacheCreationInputTokenCost *float64 `json:"cache_creation_input_token_cost,omitempty"`
|
|
CacheReadInputTokenCost *float64 `json:"cache_read_input_token_cost,omitempty"`
|
|
CacheCreationInputTokenCostAbove200kTokens *float64 `json:"cache_creation_input_token_cost_above_200k_tokens,omitempty"`
|
|
CacheReadInputTokenCostAbove200kTokens *float64 `json:"cache_read_input_token_cost_above_200k_tokens,omitempty"`
|
|
CacheCreationInputTokenCostAbove1hr *float64 `json:"cache_creation_input_token_cost_above_1hr,omitempty"`
|
|
CacheCreationInputTokenCostAbove1hrAbove200kTokens *float64 `json:"cache_creation_input_token_cost_above_1hr_above_200k_tokens,omitempty"`
|
|
CacheCreationInputAudioTokenCost *float64 `json:"cache_creation_input_audio_token_cost,omitempty"`
|
|
CacheReadInputTokenCostPriority *float64 `json:"cache_read_input_token_cost_priority,omitempty"`
|
|
|
|
// Costs - Image
|
|
InputCostPerImage *float64 `json:"input_cost_per_image,omitempty"`
|
|
InputCostPerPixel *float64 `json:"input_cost_per_pixel,omitempty"`
|
|
OutputCostPerImage *float64 `json:"output_cost_per_image,omitempty"`
|
|
OutputCostPerPixel *float64 `json:"output_cost_per_pixel,omitempty"`
|
|
OutputCostPerImagePremiumImage *float64 `json:"output_cost_per_image_premium_image,omitempty"`
|
|
OutputCostPerImageAbove512x512Pixels *float64 `json:"output_cost_per_image_above_512_and_512_pixels,omitempty"`
|
|
OutputCostPerImageAbove512x512PixelsPremium *float64 `json:"output_cost_per_image_above_512_and_512_pixels_and_premium_image,omitempty"`
|
|
OutputCostPerImageAbove1024x1024Pixels *float64 `json:"output_cost_per_image_above_1024_and_1024_pixels,omitempty"`
|
|
OutputCostPerImageAbove1024x1024PixelsPremium *float64 `json:"output_cost_per_image_above_1024_and_1024_pixels_and_premium_image,omitempty"`
|
|
OutputCostPerImageAbove2048x2048Pixels *float64 `json:"output_cost_per_image_above_2048_and_2048_pixels,omitempty"`
|
|
OutputCostPerImageAbove4096x4096Pixels *float64 `json:"output_cost_per_image_above_4096_and_4096_pixels,omitempty"`
|
|
OutputCostPerImageLowQuality *float64 `json:"output_cost_per_image_low_quality,omitempty"`
|
|
OutputCostPerImageMediumQuality *float64 `json:"output_cost_per_image_medium_quality,omitempty"`
|
|
OutputCostPerImageHighQuality *float64 `json:"output_cost_per_image_high_quality,omitempty"`
|
|
OutputCostPerImageAutoQuality *float64 `json:"output_cost_per_image_auto_quality,omitempty"`
|
|
// Costs - Audio/Video
|
|
InputCostPerAudioToken *float64 `json:"input_cost_per_audio_token,omitempty"`
|
|
InputCostPerAudioPerSecond *float64 `json:"input_cost_per_audio_per_second,omitempty"`
|
|
InputCostPerSecond *float64 `json:"input_cost_per_second,omitempty"`
|
|
InputCostPerVideoPerSecond *float64 `json:"input_cost_per_video_per_second,omitempty"`
|
|
OutputCostPerAudioToken *float64 `json:"output_cost_per_audio_token,omitempty"`
|
|
OutputCostPerVideoPerSecond *float64 `json:"output_cost_per_video_per_second,omitempty"`
|
|
OutputCostPerSecond *float64 `json:"output_cost_per_second,omitempty"`
|
|
|
|
// Costs - Other
|
|
SearchContextCostPerQuery *float64 `json:"search_context_cost_per_query,omitempty"`
|
|
CodeInterpreterCostPerSession *float64 `json:"code_interpreter_cost_per_session,omitempty"`
|
|
}
|
|
```
|
|
|
|
## Usage in Plugins
|
|
|
|
The Model Catalog is designed to be shared across all Bifrost plugins, providing consistent model information and validation logic for governance, load balancing, and other routing mechanisms.
|
|
|
|
<Note>
|
|
**Governance & Load Balancing**: Both plugins delegate model validation to the Model Catalog's `IsModelAllowedForProvider` method, ensuring consistent handling of cross-provider scenarios and provider-prefixed allowed models. See [Provider Routing](/providers/provider-routing) for configuration examples.
|
|
</Note>
|
|
|
|
### Initialization
|
|
In Bifrost's gateway, the `ModelCatalog` is initialized once at the start and shared across all plugins:
|
|
|
|
```go
|
|
import "github.com/maximhq/bifrost/framework/modelcatalog"
|
|
|
|
// Initialize model catalog with config store and logger
|
|
modelCatalog, err := modelcatalog.Init(context.Background(), &modelcatalog.Config{}, configStore, logger)
|
|
if err != nil {
|
|
return fmt.Errorf("failed to initialize model catalog: %w", err)
|
|
}
|
|
```
|
|
|
|
### Basic Cost Calculation
|
|
Calculate costs from a Bifrost response:
|
|
|
|
```go
|
|
// Calculate cost for a completed request
|
|
cost := modelCatalog.CalculateCost(
|
|
result, // *schemas.BifrostResponse
|
|
nil, // *PricingLookupScopes (nil = no scoped overrides)
|
|
)
|
|
|
|
logger.Info("Request cost: $%.6f", cost)
|
|
```
|
|
|
|
### Unified Cost Calculation
|
|
`CalculateCost` is the single entry point for all cost calculations. It handles all request types, semantic cache billing, and tiered pricing automatically:
|
|
|
|
```go
|
|
// CalculateCost handles all cost scenarios including cache-aware pricing
|
|
cost := modelCatalog.CalculateCost(result, nil) // *schemas.BifrostResponse, *PricingLookupScopes
|
|
|
|
// Cache hits return 0 for direct hits, embedding cost for semantic matches
|
|
// Cache misses return base model cost + embedding generation cost
|
|
// Returns 0.0 if pricing data is not found (logs a debug message)
|
|
```
|
|
|
|
### Model Discovery
|
|
The `ModelCatalog` provides several methods to query for model and provider information.
|
|
|
|
#### Get Models for a Provider
|
|
Retrieve a list of all models supported by a specific provider.
|
|
```go
|
|
openaiModels := modelCatalog.GetModelsForProvider(schemas.OpenAI)
|
|
for _, model := range openaiModels {
|
|
logger.Info("Found OpenAI model: %s", model)
|
|
}
|
|
```
|
|
|
|
**Thread-safe**: Uses read lock for concurrent access.
|
|
|
|
#### Get Providers for a Model
|
|
Find all providers that offer a specific model, including cross-provider resolution.
|
|
|
|
```go
|
|
gpt4Providers := modelCatalog.GetProvidersForModel("gpt-4o")
|
|
for _, provider := range gpt4Providers {
|
|
logger.Info("gpt-4o is available from: %s", provider)
|
|
}
|
|
// Result: [openai, azure, groq] (includes cross-provider mappings)
|
|
```
|
|
|
|
**Cross-Provider Resolution**:
|
|
|
|
This method implements intelligent cross-provider routing logic to discover all providers that can serve a model:
|
|
|
|
1. **Direct Match**: Checks each provider's model list in `modelPool` for the exact model name
|
|
2. **OpenRouter Format**: For models found in other providers, checks if `provider/model` exists in OpenRouter
|
|
- Example: `claude-3-5-sonnet` found in Anthropic → checks OpenRouter for `anthropic/claude-3-5-sonnet`
|
|
3. **Vertex Format**: Similar check for Vertex with `provider/model` format
|
|
4. **Groq OpenAI Compatibility**: For GPT models, checks if `openai/model` exists in Groq's catalog
|
|
5. **Bedrock Claude Models**: For Claude models, flexible matching against Bedrock's full ARN format
|
|
|
|
**Example**:
|
|
```go
|
|
providers := modelCatalog.GetProvidersForModel("claude-3-5-sonnet")
|
|
// Returns: [anthropic, vertex, bedrock, openrouter]
|
|
// Even though request was just "claude-3-5-sonnet" without provider prefix!
|
|
```
|
|
|
|
<Note>
|
|
This cross-provider logic powers Bifrost's intelligent routing capabilities. See [Provider Routing](/providers/provider-routing#the-model-catalog) for detailed examples of how this enables features like weighted routing via proxy providers.
|
|
</Note>
|
|
|
|
#### Check Model Allowance for Provider
|
|
Validate if a model is allowed for a specific provider based on an allowed models list. This method is used internally by governance and load balancing plugins.
|
|
|
|
```go
|
|
// ["*"] wildcard - uses catalog to determine support
|
|
isAllowed := modelCatalog.IsModelAllowedForProvider(
|
|
schemas.OpenRouter,
|
|
"gpt-4o",
|
|
schemas.WhiteList{"*"}, // wildcard = check catalog
|
|
)
|
|
// Returns: true (catalog knows OpenRouter supports openai/gpt-4o)
|
|
|
|
// Explicit allowedModels with provider prefix
|
|
isAllowed := modelCatalog.IsModelAllowedForProvider(
|
|
schemas.OpenRouter,
|
|
"gpt-4o",
|
|
schemas.WhiteList{"openai/gpt-4o", "anthropic/claude-3-5-sonnet"},
|
|
)
|
|
// Returns: true (strips "openai/" prefix and matches "gpt-4o")
|
|
|
|
// Explicit allowedModels without prefix
|
|
isAllowed := modelCatalog.IsModelAllowedForProvider(
|
|
schemas.OpenAI,
|
|
"gpt-4o",
|
|
schemas.WhiteList{"gpt-4o", "gpt-4o-mini"},
|
|
)
|
|
// Returns: true (direct match)
|
|
```
|
|
|
|
**Behavior**:
|
|
- **`["*"]` wildcard**: Delegates to `GetProvidersForModel` (includes cross-provider logic) — this is the "allow all via catalog" mode
|
|
- **Non-empty explicit list**: Checks for both direct matches and provider-prefixed entries
|
|
- **Empty slice (`[]string{}` / empty `schemas.WhiteList`)**: Returns `false` (deny-all) — mirrors the config deny-by-default semantics
|
|
|
|
<Note>
|
|
In `config.json` and the governance API, `allowed_models: []` (empty array) means **deny all models** (deny-by-default, v1.5.0+). The Go helper `IsModelAllowedForProvider` behaves the same way: an empty `allowedModels` slice also returns `false`. Use `["*"]` to allow all models validated through the catalog.
|
|
</Note>
|
|
- Direct: `"gpt-4o"` matches `"gpt-4o"`
|
|
- Prefixed: `"openai/gpt-4o"` matches request for `"gpt-4o"` (prefix stripped)
|
|
|
|
**Use Cases**:
|
|
- **Governance Routing**: Validate if a model request is allowed for a provider configuration
|
|
- **Load Balancing**: Filter providers based on allowed models before performance scoring
|
|
- **Virtual Key Validation**: Check if a model can be used with a specific virtual key's provider configs
|
|
|
|
<Tip>
|
|
This method is the central validation point for both governance and load balancing plugins, ensuring consistent model allowance logic across all routing mechanisms. It handles all edge cases including proxy providers (OpenRouter, Vertex) and provider-prefixed model entries.
|
|
</Tip>
|
|
|
|
#### Dynamically Add Models
|
|
You can dynamically add models to the catalog's pool from a `v1/models` compatible response structure. This is useful for providers that expose a model list endpoint.
|
|
```go
|
|
// response is *schemas.BifrostListModelsResponse
|
|
modelCatalog.AddModelDataToPool(response)
|
|
```
|
|
This is automatically done in Bifrost gateway initialization for all providers that are supported by Bifrost.
|
|
|
|
**When to use**:
|
|
- After fetching models from a provider's `/v1/models` endpoint
|
|
- When a new provider is dynamically added at runtime
|
|
- For testing with custom model lists
|
|
### Reloading Configuration
|
|
You can reload the pricing configuration at runtime if you need to change the pricing URL or sync interval.
|
|
```go
|
|
newConfig := &modelcatalog.Config{
|
|
PricingSyncInterval: 12 * time.Hour,
|
|
}
|
|
err := modelCatalog.UpdateSyncConfig(ctx, newConfig)
|
|
```
|
|
|
|
## Error Handling and Fallbacks
|
|
|
|
The Model Catalog handles missing pricing data gracefully with intelligent fallbacks:
|
|
|
|
```go
|
|
// resolvePricing resolves the pricing entry for a model, trying deployment as fallback.
|
|
func (mc *ModelCatalog) resolvePricing(provider, model, deployment string, requestType schemas.RequestType) *configstoreTables.TableModelPricing {
|
|
pricing, exists := mc.getPricing(model, provider, requestType)
|
|
if exists {
|
|
return pricing
|
|
}
|
|
// If pricing not found for model, try the deployment name
|
|
if deployment != "" {
|
|
pricing, exists = mc.getPricing(deployment, provider, requestType)
|
|
if exists {
|
|
return pricing
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// getPricing returns pricing information for a model (thread-safe).
|
|
// It implements a multi-step fallback chain:
|
|
// 1. Direct lookup by model + provider + mode
|
|
// 2. Gemini → Vertex provider fallback
|
|
// 3. Vertex "provider/model" prefix stripping
|
|
// 4. Bedrock "anthropic." prefix addition for Claude models
|
|
// 5. Responses → Chat mode fallback (at each step)
|
|
// 6. ImageEdit / ImageVariation → ImageGeneration mode fallback
|
|
func (mc *ModelCatalog) getPricing(model, provider string, requestType schemas.RequestType) (*configstoreTables.TableModelPricing, bool) {
|
|
mc.mu.RLock()
|
|
defer mc.mu.RUnlock()
|
|
|
|
mode := normalizeRequestType(requestType)
|
|
|
|
pricing, ok := mc.pricingData[makeKey(model, provider, mode)]
|
|
if ok {
|
|
return &pricing, true
|
|
}
|
|
|
|
// Provider-specific fallbacks (Gemini→Vertex, Vertex prefix strip, Bedrock anthropic. prefix)
|
|
// Each fallback also tries Responses→Chat mode if applicable
|
|
// ...
|
|
|
|
// Final fallback: Responses → Chat mode for any provider
|
|
if requestType == schemas.ResponsesRequest || requestType == schemas.ResponsesStreamRequest {
|
|
pricing, ok = mc.pricingData[makeKey(model, provider, normalizeRequestType(schemas.ChatCompletionRequest))]
|
|
if ok {
|
|
return &pricing, true
|
|
}
|
|
}
|
|
|
|
return nil, false
|
|
}
|
|
|
|
// When pricing is not found, CalculateCost returns 0.0 and logs a debug message.
|
|
// This ensures operations continue smoothly without billing failures.
|
|
```
|
|
|
|
|
|
## Cleanup and Lifecycle Management
|
|
|
|
Properly clean up resources when shutting down:
|
|
|
|
```go
|
|
// Cleanup model catalog resources
|
|
defer func() {
|
|
if err := modelCatalog.Cleanup(); err != nil {
|
|
logger.Error("Failed to cleanup model catalog: %v", err)
|
|
}
|
|
}()
|
|
```
|
|
|
|
## Thread Safety
|
|
|
|
All `ModelCatalog` operations are thread-safe, making it suitable for concurrent usage across multiple plugins and goroutines. The internal pricing data cache uses read-write mutexes for optimal performance during frequent lookups.
|
|
|
|
## Best Practices
|
|
|
|
1. **Shared Instance**: Use a single `ModelCatalog` instance across all plugins to avoid redundant data synchronization.
|
|
2. **Error Handling**: Always handle the case where pricing returns 0.0 due to missing model data.
|
|
3. **Logging**: Monitor pricing sync failures and missing model warnings in production.
|
|
4. **Cache Awareness**: Use `CalculateCost` which automatically handles cache hits/misses and embedding costs.
|
|
5. **Resource Cleanup**: Always call `Cleanup()` during application shutdown to prevent resource leaks.
|
|
|
|
The Model Catalog provides a robust, production-ready foundation for implementing billing, budgeting, and cost monitoring features in Bifrost plugins.
|